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\  1  ABSTRACT 


This  manual  is  a  working  document  to  assist  in  evaluating  a  variety 
of  information  processing  activities.  A  generalized  procedure  is  outlined 
to  assess  the  capability  of  an  information  system  to  meet  functional,  user- 
oriented  requirements.  The  authors  propose  a  composite  overview  which  will 
be  the  basis  of  a  systematic  approach  to  information  system  evaluation. 
S.ress  is  placed  upon  identification  of  tangible  benefits  and  measures  of 
effectiveness.  Refinement  and  standardization  of  the  approach  arc  antici¬ 
pated  on  the  basis  of  experience  in  applying  its  phases  to  systems  operating 
under  differing  circumstances. 
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Foreword 


This  manual  is  a  working  document  based  upon  experiences 
of  a  number  of  people  in  evaluating  a  variety  of  information 
processing  activities.  The  authors  propose  a  composite  over¬ 
view  which  will  be  the  basis  of  a  systematic  approach  to  in¬ 
formation  system  evaluation.  The  procedures  outlined  herein 
are  intended  to  be  applied  to  several  operational  systems  and 
the  results  analyzed  to  determine  the  value  and  cost  associated 
with  each  step  under  differing  circumstances.  Refinement  of 
the  approach  and  a  revised  manual  are  anticipated  after  expe¬ 
rience  is  gained  from  these  evaluation  studies. 
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I.  Introduction 


For  years  computers  and  computer-oased  information  systems 
have  been  sold  on  the  strength  of  intangible,  and  often  illusory, 
benefits.  Prevailing  economic  conditions  coupled  with  common 
underestimates  of  resource  and  time  requirements  have  engendered 
recently  an  era  of  greater  discrimination  by  managers  and  computer 
users-  Formation  of  profit  or  cost  centers  for  information  pro¬ 
cessing  services,  application  of  capital  authorisation  criteria  for 
system  development,  and  closer  scrutiny  of  operating  expenses  for 
computing  and  information  processing  have  resulted.  Attempts  to. 
evaluate  the  effectiveness  of  systems  have  been  thwarted  by  diffi¬ 
culty  in  identifying  and  quantifying  tangible  benefits  resulting 
from  operation  of  the  system.  This  stems  from  lack  of  definition 
of  system  objectives  in  functional  terms,  of  standardization  in 
applying  data  processing  techniques,  and  of  formalization  in  methods 
for  evaluation.  The  latter  area  is  of  particular  interest  today, 
because  current  practice  becomes  relatively  less  adequate  as  infor¬ 
mation  systems  deal  increasingly  with  more  complex,  integrated  func¬ 
tions.  This  manual  will  assume  the  folic  wing  general  description: 

•  Information  System  —  A  set  of  interrelated  rules  and  pro¬ 
cedures  for  processing  data  into  information  in  order  to  get 
or  control  action. 

A  substantial  amount  of  empirical  study  is  directed  toward 
performance  measurement  of  computer  systems.  Emphasis  therein  is 
placed  on  speed  of  computer  operations,  throughput  of  the  computer, 
capacity  utilization  of  processing  components,  and  response  time  to 
the  user.  These  are  useful  in  comparing  alternative  methods  of 
achieving  given  results  or  in  assuring  timeliness  of  output.  Com¬ 
puter  specialists,  such  as  system  analysts  and  programmers,  are 

likely  to  appreciate  and  strive  for  improved  performance  in  these 
1,  4,  5,  8 

areas. 

The  customer  of  information  processing  services,  however,  is 
more  concerned  v't-h  nualitv  of  system  output.  Quality  is  related 


to  accuracy  of  input,  reliability  of  processing,  and  appropriate¬ 
ness  of  output:  that  is,  insuring  adequate,  but  not  excessive, 
reports  according  to  need  and  the  situation.  The  degree  of  detail 
in  output  and  frequency  of  reporting  varies  with,  among  other 
things,  the  level  of  activity  being  served.  General  classifica¬ 
tions  of  these  levels  are  shown  in  Table  1-1.  Various  forms  of 
management  oecision-making  are  required  by  these  levels  of  activ¬ 
ity.7  Decision-oriented  systems  normally  include  attributes  of 
several  activity  especially  those  involving  tactical  and 

operational  aspects.  The  benefit  parameters  of  computer-based 
systems  must  be  related  to  these  functional  activity  descriptions 
in  order  to: assess  the  effect  of  information  on  the  behavior  of 
the  customer,  or  system  user.  This  latter  area  is  of  major  con¬ 
cern  because  it  relates  to  impact  on  management  decision-making 
and  short  range  planning  rather  than  the  narrower  horizon  of  com¬ 
puter  performance  in  gathering  and  manipulating  data.  This  em¬ 
phasis  also  offers  greater  challenge  to  analyze  information  system 
effectiveness  in  light  of  dynamically  changing  personal  and  or¬ 
ganizational  requirements  for  information. 

In  this  sense,  information  is  defined  as  a  response  to  user 
demand  that  is  timed  and  structured  to  be  useful  in  decision¬ 
making  or  control  tasks  (as  opposed  to  an  accumulation  of  facts, 
or  data).  This  suggests  a  need  to  isolate  the  results  of  deci¬ 
sions  from  associated  input  to  decision-makers  in  order  to  deter- 

2  3  6 

mine  the  quality  of  information  provided  by  the  system.  *  ’ 

Messages  which  convey  information,  have  the  following  attri¬ 
butes  relative  to  the  problem  at  hand ■ 

•  Timeliness 

•  Accuracy 

•  Reitvancy 

•  Sufficiency 

«  Conciseness 

•  Discovery 
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Execution  Performance  of  assigned  functional  tasks  by  ap-  Reports  of  completed  tasks 

plying  available  resources.  Concerns  established  become  basic  input  to  the  in- 

procedures  necessary  to  accomplish  an  assignment  formation  system.  Includes 

or  process.  record-keeping  and  conditioned 

reactions  to  transactions. 


The  latter  is  particularly  important  because  it  assures  that  the 
recipient  learns  something  of  which  he  was  unaware  before  receiv¬ 
ing  the  message. 

The  results  of  studying  and  developing  evaluation  procedures 
for  existing  information  systems  provide  insight  about  assessing 
the  effectiveness  of  complex,  multivariable  situations.  Normally 
an  information  system  will  include  one  or  more  of  the  processing 
characteristics  outlined  in  Table  1-2.  Study  should  indicate  the 
blend  of  characteristics  existing  or  planned  which  would  satisfy 
identified  functions  and  perform  specific  tasks.  It  should  then 
be  possible  ';o  extrapolate  tnose  results  to  forecast  the  demand  for 
information  services  ar.d  to  propose  the  most  effective  and  econom¬ 
ical  means  of  providing  same. 

A  commonly  recognized  fault  limiting  the  success  of  informa¬ 
tion  systems  is  the  lack  of  customer  or  user  participation  in  the 
definition  of  system  requirements.  These  should  be  stated  in  a 
formal  document  oriented  to  terminology  and  functions  of  the  opera¬ 
tional  gioup  to  be  served.  The  customer  should  provide  about  half 
of  the  effort  in  the  analysis  and  feasibility  phases  which  should 
culminate  in  a  customer  approved  report  on  requirements  which  in¬ 
cludes:  (See  Appendix  B  for  detail.) 

•  Mission  or  problem  statement 

•  System  objectives 

•  Output  and  report  definition 

•  Input  description 

•  File  definition 

•  Controls  and  tests 

-  Schedule  constraints 

•  Cost  analysis 

•  Project  plan 

It  is  assumed  that  the  accuracy  of  the  cost  analysis  and  the 
effectiveness  of  the  project  plan  are  assessed  best  during  and  im¬ 
mediately  after  the  development  of  the  system.  This  set  of  user 

requirements  then  becomes  the  primary  sou  e  for  evaiuatina  effcc- 

r  .  ,  _  „  b,  10,  11 

tiveness  of  information  system  operation. 
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Table  1-2 

Processing  Characteristics 


Planning 


Management 


Communication 


Data  Base 


Monitoring 


Reference 


Scientific 


Models  to  project  and  forecast  based  on  historical 
trends  and  estimates  of  the  future.  Normally  re¬ 
quires  a  representative  sample  data  base  rather 
chan  live  operational  data.  Interactive  mode  of 
operation  is  highly  desirable. 

analysis  of  the  utilization  of  resources  by  func¬ 
tional  managers.  Requires  complete  actual  data. 
Mode  of  operation  is  usually  batch -oriented  unless 
decisions  are  related  to  short  cycle  period. 

Transmission  of  messages  among  remote  sites.  Re¬ 
quires  proper  distribution  of  input  without  inter¬ 
pretation  or  alteration  of  content.  Normally  re¬ 
quires  on-line  processing  with  store  and  forward 
capability. 

Collection  of  data  and  storing  in  computer-based 
files  for  future  access  to  content  or  facts.  Re¬ 
quires  frequent,  if  not  continuous,  input  of  data 
from  a  variety  of  sources.  Normally  requires  on¬ 
line  inquiry  mode  for  effective  delivery  of  output. 

Gathering  and  analysis  of  data  from  a  specific  pro¬ 
cess  or  operation  on  a  continual  basis.  Normally 
involves  on-line  capturing  of  digital  and/or  analog 
data  and  immediate  feedback  to  control  the  opera¬ 
tion. 

Retrieval  of  documents  or  references  thereto,  often 
based  on  material  stored  in  microform,  technical 
papers,  and  other  media  not  computer-compatible. 
Initial  reference  is  often  accomplished  by  on-line 
computer  operation.  Delivery  of  the  desired  input 
is  normally  an  off-line,  batch  oriented  library 
process. 

Algorithm  oriented  processing  related  tc  calcula¬ 
tion  for  analysis  or  estimation  of  operational 
data.  Includes  statistical  and  engineering  cal¬ 
culation.  Processing  is  normally  done  in  batch 
mode  but  input  may  be  collected  on-line  or  ag¬ 
gregated  for  subsequent  processing. 
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II.  Preparation  for  an  Evaluation  Study 


Purpose  of  Study 

For  an  evaluation  study  to  be  successful,  it  is  necessary 
that  management  be  willing  to  act  upon  recommendations  of  the 
study  finding.  Also,  it  should  be  likely  that  resources  to  ef¬ 
fect  necessary  changes  will  be  available  to  implement  any  deci- 

1  3 

sions  resulting  from  the  evaluation.  ’  The  persons  authorizing 
the  study  and  receiving  the  findings  must  have  purview  over  the 
system  operation  and  the  functional  activities  served  by  the  sys¬ 
tem.  Motivation  to  assess  system  effectiveness  generally  stems 
from: 

—  Need  to  diagnose  problems  in  an  existing  system  (including 
cost  of  operation). 

—  Dissatisfaction  expressed  by  users  of  a  system. 

—  Proposal  to  modify  or  revise  a  system. 

—  Flans  to  implement  a  new  system. 

—  Scheduled  event  for  audit  or  evaluation. 

In  any  case,  the  objectives  of  the  evaluation  study  need  to  be 
carefully  delineated  in  order  to  assure  that  the  analysis  achieves 
management  purposes. 

Evaluaticn  should  consider  the  expected  life  of  the  system 

but,  unlike  some  life  cycle,  analvsis,  the  time  value  of  costs  and 

19 

benefits  must  be  considered  at  appropriate  rates  of  return. 
Particular  attention  should  be  given  to  identifying  possible  un¬ 
intended  benefits  or  siJe  effects  rather  than  concentrating  on 
initial  statements  of  objectives.  A  number  of  other  considerations 
are  important  to  the  success  of  such  a  study,  namely: 

—  User  involvement  in  che  evaluation. 

—  Written  charge  to  the  study  team  to  clarify  purpose  and 
organizational  relationships. 

—  Authority  from  management  with  purview  over  full  scope  of 
the  systems. 

—  Timing  the  study  for  representative  periods  of  activity. 
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—  Review  of  documentation,  but  validate  its  accuracy  and 
completeness  by  us_  other  sources. 

—  Review  of  prior  evaluation  or  audit  reports  (By  whom, 
findings,  resulting  action?) 

—  Avoid  the  study  if  it  is  motivated  by  coercion  or  if 
management  is  indifferent. 

—  Phrase  reports  in  the  language  of  management,  not  system 
terminology. 

—  Design  the  system  initially  to  provide  performance  data 
as  a  by-product  of  normal  system  operation. 

Above  all,  the  need  for  evaluation  should  be  apparent  and  follow¬ 
up  action  possible  or  the  study  is  almost  surely  due  for  the 
archives . 

Organization  of  the  Study  Team 

Because  of  the  diverse  interests  and  broad  implications  of 
most  information  systems,  evaluation  should  be  conducted  by  a  team. 
Representatives  of  major  functional  activities  served  by  the  sys- 
tem(s)  and  practitioners  in  data  processing  should  be  the  principal 
constituents.  Consultants  often  supply  objectivity  and  breadth  of 
experience  to  the  study.  Depending  upon  study  emphasis  and  ob¬ 
jectives,  it  may  be  desirable  to  include  an  accountant,  an  orga¬ 
nizational  behaviorist,  or  other  specialists  on  the  team.  The 
role  of  the  system  designer,  development  contractor,  and  computer 
operations  should  be  minimized  to  that  of  sources  of  evidence 

rather  than  team  members.  In  general,  four  or  five  people  would 

3  5  41 

be  assigned  responsibility  to  conduct  the  study.  * 

Leadership  of  the  evaluation  team  should  come  from  a  con¬ 
sultant  or  from  an  operational  manager  whose  activities  are 
directly  affected  by  the  system.  In  either  case,  the  individual 
must  have  objectivity  and  also  have  sufficient  influence  to  ini¬ 
tiate  short  term  improvements  or  changes  during  the  study  that 
are  proven  necessary.  A  secretary  or  coordinator  should  be  ap¬ 
pointed  to  act  as  a  repository  for  collected  data  and  interviews, 
to  record  notes  and  decisions,  and  to  prepare  and  disseminate  re¬ 
ports.  The  study  leader  normally  arranges  schedules  and  monitors 
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progress  of  the  study.  It  is  important  for  all  members  of  the 

team  to  establish  and  maintain  rapport  with  functional  management 

41  44 

requirements  throughout  the  study.  * 

Study  Plan 

C  ce  the  objective,  scope,  and  team  membership  have  been  de¬ 
fined  and  published,  it  is  possible  to  develop  a  study  plan.  The 
major  tasks  to  be  completed  in  the  study  plan  are  discussed 
briefly.  *  * 

Task  1:  ,  Orientation.  The  first  task  is  to  indoctrinate  both 
management  and  the  study  team.  Higher  levels  of  management 
must  be  oriented  to  the  importance  of  system  users,  to  the 
current  state  of  the  art  in  computing,  and  to  formal  descrip¬ 
tion  of  their  functional  responsibilities.  The  study  team 
needs  to  consider  the  boundaries  of  the  system,  policy  con¬ 
straints,  and  assumptions  necessary  to  define  the  problems 
and  issues  before  them.  Background  of  system  requirements, 
related  prior  studies,  and  existing  performance  standards 
also  heln  describe  the  environment  in  which  the  evaluation 
will  be  conducted.  Particular  attention  should  be  given  to 

identification  of  existing  base  lines  for  performance  and 
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measures  of  effectiveness.  *  ’  If  the  latter  are  not 

already  established,  they  must  be  developed  during  Task  6. 

Task  2:  Selection  of  phases.  The  generalized  procedure  in 
part  III  of  this  manual  outlines  a  number  of  potential  phases 
for  the  study.  Each  phase  must  be  considered  to  determine 
its  relevancy  to  the  authorized  study.  The  objectives  of  the 
study  and  likely  availability  of  staff  time  and  resources 
will  suggest  the  degree  of  detail  for  the  study.  For  each 
possible  phase,  selection  must  be  made  according  to  the 
Activity  Levels  (Table  1-1)  and  Processing  Characteristics 
(Table  1-2)  associated  with  the  system(s)  to  be  assessed. 

Task  3:  Selection  of  methods.  For  each  phase  that  is  deemed 
appropriate  for  the  study,  there  can  be  a  variety  of  methods 
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Table  II -1 

Methods  of  Investigation 
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Views :  Subjective  nnd  Qualitative  variables  with  user 
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employed.  In  some  steps,  the  comparison  of  costs  among 
alternatives  can  be  nil  or  greater.  Table  II-l  summarizes 
the  most  commonly  used  methods  and  issues  related  to  them. 
Cost  considerations  may  eliminate  a  phase  from  the  study  or 
suggest  a  more  limited  estimate  of  the  situation.  The  ref¬ 
erences  at  the  end  of  this  section  refer  briefly  to  basic 
concepts  of  system  analysis.  (See  Table  II-l).  Further, 
they  provide  some  overview  of  the  evaluation  process  and 
examples  of  gaming  and  Delphi  techniques.  Both  the  latter 
can  be  modified  to  assist  analysis  of  highly  subjective 
areas  and  to  minimize  the  bias  resulting  from  orally  ex¬ 
pressed  opinions.  Each  requires  special  skills  to  develop 
and  conduct  investigation  by  the  method. 

Particular  attention  should  be  given  to  getting  candid 
responses  in  the  various  methods.  Careful  preparation  of 
managers  for  interviews  may  promote  answers  which  reflect 
how  the  system  should  work  rather  than  how  it  does.  Im¬ 
promptu  sessions  tend  to  elicit  normal  reactions  and  measure 
personal  understanding  rather  than  encouraging  staff  brief¬ 
ings  which  reflect  intent  or  composite  views. 

Task  4;  Establishment  of  schedule  and  priorities.  The  cost/ 
benefit  estimates  for  each  phase  will  suggest  relative  pri- 
orities.  The  number  of  personnel  assigned  to  the  study 
team  and  the  selected  methods  are  the  basis  for  developing  a 
pattern  of  investigation  and  analysis.  Restrictions  on  time 
to  complete  the  evaluation  will  impact  this  pattern  and, 
perhaps,  will  necessitate  revision  of  selected  phases  or 
methods.  A  study  will  normally  be  4-7  weeks  in  duration. 
Basic  studies  may  require  only  a  week,  but  major  systems 
with  ill-defined  objectives  may  necessitate  six  months  or 
more  of  elapsed  ^’me."' 

Task  5;  Estimation  of  resource  needs.  Often  the  size  of 
the  study  team  and  authorized  expenses  are  fixed.  Thus,  the 
major  resource  to  be  identified  is  the  demand  for  na nagement 
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and  system  user  time.  Sometimes  either  the  complexity  or 
volume  of  study  effort  will  require  supplementary  team 
assignments.  The  proportion  and  timing  of  participation 
by  each  team  member  should  be  allocated  to  the  proposed 
schedule.  Technical  and  support  assistance,  particularly 
that  related  to  selected  methods,  should  also  be  included. 

Task  6:  Affirmation  by  management.  It  is  important  to  re¬ 
inforce  management  interest  and  to  ensure  their  understanding 
of  the  study.  The  objectives,  time  schedule,  format,  and 
resource  needs  should  be  presented  for  review.  Particular 
attention  must  be  given  to  the  contacts  and  interactions 
expected  between  the  study  team  and  operational  management. 
Groundwork  is  laid  to  overcome  possible  resistance  to  change 
and  anticipated  personal  conflicts.  Agreement  in  advance  of 
the  study  proper  identifies  the  groups  that  will  be  involved, 
approves  the  study  plan,  and  reaffirms  the  study  team  au¬ 
thority. 

Task  7:  Conduct  of  phases.  The  conduct  of  each  selected 
phase  is  discussed  in  part  III.  Instruments  for  collecting 
data  are  designed;  treatment  of  data  is  planned,  and  criteria 
for  analysis  are  designated.  Data  are  gathered,  aggregated 
and  stored  using  the  various  methods  and  sources.  The  study 
team  analyzes  input  and  debriefing  sessions  are  held  to  keep 
management  and  team  personnel  aware  of  progress  and  interim 
findings.  The  results  of  each  phase  are  documented  as  ex¬ 
hibits  to  the  final  report. 

Task  8:  Dissemination  of  findings.  Principal  results  of  the 
study  should  be  reported  to  the  affected  management  orally. 

This  may  be  done  by  meetings,  talks,  or  highly  visual  pre¬ 
sentations.  A  written  follow-up  should  be  presented  by  a 
forma],  report,  a  series  of  memoranda,  or  other  hard  copy  media. 
The  content  should  he  expressed  in  management  or  user  vernac¬ 
ular  and  limited  to  important  issues  requiring  attention. 
Exhibits  and  background  material  should  be  maintained  on  file 


by  the  study  team.^'  ^  ^  Ihe  written  report  should  contain: 

i 

—  General  information 

•  Purpose  and  goals  of  evaluation 

•  Team  composition 

•  Methods  used 

4  Timing 

•  Areas  assessed 

•  Summary  of  meetings  and  oral  presentations 
—  Summary  of  findings 

•  Significant  accomplishments 

•  Adverse  performance  areas 

•  Recommended  action(s) 

—  Management  rebuttal  or  comment 
—  Study  team  rejoinder 

Recurring  Audits 

Periodic  evaluations  should  be  conducted  on  the  basis  of 
the  expected  system  life,  which  is  rarely  longer  than  3-5 
years.  Each  study  will  determine  the  current  effectiveness 
of  the  system  and  predict  its  capability  to  handle  expected 
growth  and  change.  In  addition,  the  recurring  audits  will 
measure  results  of  implementing  recommended  action  which  stems 
from  prior  evaluation  findings ^  and  will  appraise  the  system 
against  alternative  ways  to  support  the  same  functional  re¬ 
quirements  . 
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III.  Generalized  Evaluation  Procedure 


The  evaluation  of  information  systems  involves  six  major 
areas  of  study  and  analysis.  The  first  three  phases  (Al-3)  de¬ 
scribe  the  organizational  functions  performed  at  the  various  lev¬ 
els  of  activity,  the  remaining  three  phases  (Bl-3)  relate  to 
identification  and  description  of  the  information  system(s)  which 
ser'-e  these  functions.  An  outline  of  the  procedure  follows: 

A.  Characterize  the  function  served 

1.  Define  Organization  Mission 

2.  Describe  F  notional  Performance 

3.  Identify  T  .ternative  Courses  of  Action 

B.  Assess  the  current  state  of  the  information  system 

1.  Define  the  system(s)  characteristics 

2.  Audit  economics  of  the  information  system 

3.  Measur'  information  system  activity 

Each  of  these  areas  must  be  assessed  as  to  its  importance  in 
a  given  evaluation  study.  Thus,  the  actual  study  plan  must  be 
tailored  from  the  generalized  procedures  in  order  to  achieve  study 
objectives.  The  contribution  of  each  area  must  be  considered  in 
light  of  the  kind  of  system  under  study,  the  level(s)  of  activity 
involved,  and  the  methods  of  analysis  to  be  employed.  The  Purpose 
amplifies  the  contribution  of  this  step  to  an  evaluation  study. 
Circumstances  under  which  the  step  would  be  appropriate  are  indi¬ 
cated  under  Selection.  Possible  methods  of  Investigation  and  sug¬ 
gestions  about  Conduct  of  each  facet  of  the  study  are  also  dis¬ 
cussed  at  some  length.  Information  for  further  study,  charts  and 
reports  produced  in  the  step  are  described  briefly  in  the  Result 
section. 

A .  Characterize  the  Function  Served 

1.  Define  Organization  Mission.  This  is  a  clear  statement  of 
the  mission  and  expected  life  of  functions  performed  by  the 
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customer  organization  to  accomplish  the  mission.  The  nature 
of  information  services  used  and  interactions  with  ether 
groups  in  handling  data  are  important  parts  of  the  description. 
Particular  attention  should  be  given  to  identifying  environ¬ 
mental  influences  on  the  functions  performed.  It  is  particu¬ 
larly  important  to  note  changes  in  assumptions  or  requirements 
which  would  obsolete  documentation  about  a  pertinent  system 
requirement. 


a.  Major  Functions 

•  Purpose:  To  explore  the  character  and  expected  life 
of  functions  performed  by  the  customer  organization. 

Changes  and  trends  in  the  functions  performed  should  be 
carefully  noted,  as  these  may  not  be  reflected  in  the  doc¬ 
umentation  on  policy  and  procedures. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant  and  necessar 
Relcvjnt  and  necessary 
Not  relevant 
Relevant  and  necessary 
Relevant  and  necessary 
Possiole  for  tactical  activity 
Relevant  in  some  situations 


Activity  Levels  —  Strategic  and  tactical 

Methods  —  Structured  or  depth  interview;  group 
consensus 

•  Conduct:  The  evaluation  team  meets  upper  management 
for  an  overall  orientation  to  the  organization  and  its 
goals.  Major  functional  components  of  the  organization  and 
the  corresponding  management  personnel  are  identified. 
Members  of  the  evaluation  team  then  conduct  interviews  with 
managers  at  the  tactical  level,  soliciting  individual  views 
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on  the  nature  of  each  function  and  its  expected  life.  A 
structured  interview  should  require  from  about  half  an 
hour  per  manager  to  a  maximum  duration  of  one  hour  and  a 
half.  An  open-ended  depth  interview  would  generally  re¬ 
quire  at  least  half  a  day,  preferably  being  conducted  in 
several  sittings.  If  functions  are  not  well  defined  or 
established,  it  may  be  necessary  to  resolve  subjective 
opinions  using  more  elaborate  methods  to  develop  a  con¬ 
sensus  . 

The  evaluation  team  should  meet  at  frequent  inter¬ 
vals  during  this  interview  phase  to  cross-check  the  func¬ 
tions  served  and  their  interrelationships.  Inconsisten¬ 
cies  should  be  resolved  by  going  back  to  the  managers  for 
clarification.  If  key  subfunctions  are  recognized  at 
this  time,  the  evaluation  team  may  then  decide  to  include 
subordinate  managers  in  the  interview  procedure.  Schedules 
should  be  modified  accordingly. 

Following  is  a  checklist  of  questions  to  guide  the 
evaluation  in  conducting  the  interview: 

(1)  What  are  the  major  functions  in  each  area  of 
responsibility? 

(2)  How  have  these  changed  over  the  past  several 
years?  (Use  time  frame  of  current  information  sys¬ 
tems) 

(3)  What  trends  are  anticipated  in  performance  of 
these  functions? 

(4)  What  is  the  expected  life  of  each  function 
within  the  organization? 

•  Result:  The  evaluation  team  should  reach  general 
agreement  on  the  major  functions  performed  by  the  organiza¬ 
tion  so  that  the  functional  components  of  an  information 
flow  diagram  can  be  sketched  as  input  to  later  steps. 

b.  Information  Support  Requireu  or  Provided 

•  Purpose:  To  examine  the  general  nature  of  information 
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support  required  or  provided  in  the  course  of  performing 
major  functions. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Not  relevant 
Relevanc  and  necessary 
Relevant  and  necessary 
Advisable  for  tactical  activity 
Often  relevant  and  advisable 


Activity  Levels  —  Strategic  and  tactical 

Methods  —  Structured  interview,  questionnaire,  or 
observation 

•  Conduct:  If  possible,  this  should  be  investigated 
during  the  process  of  defining  user  functions.  The  struc¬ 
tured  interview  is  preferred  for  this  phase,  with  the  fol¬ 
lowing  checklist  of  questions  to  guide  the  interviewer: 

(1)  Describe  the  activities  which  are  dependent  on 
information  service  support  to  perform  each  function. 

(2)  Describe  the  activities  which  involve  providing 
information  service  support  to  other  group(s). 

(3)  Describe  the  nature  of  each  such  information 
service. 

(a)  Normal  use  of  the  service. 

(b)  Special  uses  of  the  service.  Under  what 

conditions? 

(c)  Schedule  of  usage  and  correspondence  to 

events . 

(d)  Consequences  of  failure. 

(e)  Required  ior  how  long? 

"In-basket"  analysis  is  often  useful  in  determining 
interaction  of  information  services.  This  entails  observa¬ 
tion  of  the  concent  of  material  received  and  disseminated 
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by  a  manager  in  order  to  determine  the  frequency  and 
nature  of  information  from  and  provided  to  other  groups. 
Questionnaires  may  be  utilized  to  conserve  time  and/or 
expense. 

•  Result:  The  evaluation  team  should  reach  general 
agreement  on  the  information  used  in  conjunction  with 
performance  of  functions.  Major  information  flows  should 
be  described  in  matrix  or  network  format. 

c  •  Environmental  Influences 

•  Purpose:  To  explore  environmental  influences  on 
groups  within  the  organization;  especially  those  restric¬ 
tions  which  limit  effective  use  of  information  support 
services . 

•  Selection: 


Characteristic: 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitoring 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Sometimes  relevant 
Frequently  relevant 
Frequently  relevant 
Relevant  at  tactical  level 
Frequently  relevant 


Activity  Levels  —  Strategic,  tactical  and  opera¬ 
tional 


Methods  —  Delphi  study,  questionnaire,  depth  inter 
view 


•  Conduct;  External  influences  should  be  elicited 
during  the  process  of  defining  user  functions  and  informa¬ 
tion  support.  Other  methods  involve  substantial  expense  to 
get  more  than  cursory  views.  This  phase  deals  with  future 
events  and  its  speculative  nature  frequently  necessitates  a 
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Delphi  study.  Since  there  is  little  precedent  for  deter¬ 
mining  effects  of  environmental  factors,  an  open-ended 
depth  interview  of  specialists  in  several  management  and 
staff  areas  is  appropriate.  A  checklist  of  questions 
follows : 

(1)  What  external  problems  or  restrictions  limit  ef¬ 
fect!  '/p  use  of  information  and  information  systems? 

lechnology 
Resources  Available 

Experience  and  Capability  of  Personnel 

Market  and  Competition 

Management  Policy 

Political  and  Governmental  Issues 

Economic  Factors  (GNP,  employment,  productivity, 

price  and  wages) 

Demographic  and  Social  Factors 
Public  Interest  and  Acceptance 

(2)  In  reference  to  the  above  factors,  what  changes  or 
trends  have  altered  use  of  information  and  information 
systems  in  the  recent  past? 

(3)  What  changes  or  trends  are  anticipated  in  the 
foreseeable  future? 

(4)  Alternatively,  does  use  of  information  and/or  in¬ 
formation  systems  exert  any  positive  or  negative  in¬ 
fluence  on  these  factors? 

The  evaluation  team  may  also  decide  to  seek  expert 
opirion  from  outside  the  organization  to  help  weigh  the 
affect  of  environmental  factors  .  This  is  especially 
important  in  resolving  controversial  issues  I’aised  in 
this  step. 

A  questionnaire  will  generally  be  the  most  cost- 
effective  way  to  seek  comprehensive  opinions;  although 
a  full  Delphi  study  may  be  warranted  in  instances  where 
environmental  factors  play  an  unusually  significant 
role,  questions  should  follow  the  general  interview 
outline,  but  should  be  tailored  to  emphasize  coverage 
of  important  areas  revealed  in  earlier  interviews. 
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•  Result:  This  phase  should  produce  a  clear  state¬ 
ment  of  environmental  factors  and  their  possible  effects 
on  information  usage. 

2.  Describe  Functional  Performance.  The  major  tasks  and  se¬ 
quences  to  perform  functions  are  reviewed  and  developed  into 
a  scenario  of  personnel  interacting  in  activities  which  re¬ 
quire  use  of  information  services,  identifying  the  various 
classifications  of  activity  levels.  Options  and  possible 
events  related  to  functional  performance  are  recorded.  The 
nature  of  data  flow  and  the  interactions  among  people  are  re¬ 
viewed.  Often  it  will  be  necessary  to  assimilate  subjective 
opinions  about  desired  or  actual  performance.  Operational 
gaming  can  be  developed  to  observe  individual  behavior  toward 
information  when  confronted  with  functional  tasks  and  to  in¬ 
struct  in  more  effective  methods  of  information  use.  Job 
descriptions  and  procedures  manuals  are  considered  to  the 
degree  that  they  prove  accurate  and  current  to  actual  prac¬ 
tice. 


a .  Major  Tasks  and  Sequence 

•  Purpose:  To  describe  major  tasks  and  activity  se¬ 
quences  for  each  function.  These  may  be  derived  from  those 


functions  identified  in  ea 

•  Selection: 

Characteristics 

Planning 

Management 

Communication 

Data  Base 
Monitoring 
Reference 
Scientific 


■Her  phases  of  the  study. 


Rarely  relevant 

Relevant  and  necessary 

Relevant  for  execution  and 
operational  levels 

Relevant  and  necessary 

Relevant;  normally  documented 

Sometimes  relevant 

Relevant;  normally  documented 
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Activity  Levels  —  Tactical,  operational  and 
execution 

Methods  —  System  documentation,  interviews 

•  Conduct:  Any  existing  documentation  on  job  descrip¬ 
tions  and  standard  procedures  for  performing  repeated  func¬ 
tions  should  be  collected  and  reviewed  by  the  evaluation 
team.  Since  it  frequently  does  not  reflec:  actual  practice, 
it  is  also  advisable  to  question  informally  managers  as  to 
the  relevance  and  accuracy  of  this  documentation. 

If  there  are  strong  indications  that  the  documentation 
does  in  fact  deviate  significantly  from  actual  practice, 
then  interviews  should  be  conducted  in  depth.  Conversely, 
if  the  documentation  is  felt  to  be  relevant  and  accurate, 
then  the  interview  can  be  conducted  as  a  verbal  review  of 
the  documentation,  clarifying  minor  deviations  from  and 
implications  of  the  written  procedures. 

Following  is  a  checklist  of  questions  to  guide  the 
evaluator  in  conducting  the  interview: 

(1)  Describe  step  by  step  the  tasks  which  are  per¬ 
formed  to  support  previously  identified  functions. 

(?)  What  events  affect  performance  or  sequence  of 
these  tasks? 

(3)  What  options  are  possible  in  performing  these 
tasks? 

•  Result:  A  diagram  or  flow-chart  of  the  tasks  per¬ 
formed  in  sequence  should  be  developed  to  accompany  the 
verbal  descriptions  elicited  during  interviews.  External 
events  and  internal  options  which  affect  task  sequence 
should  be  depicted  as  branch  points  on  the  diagram. 

b.  Scenario  of  Information  Use 

•  Purpose:  To  review  major  tasks  and  procedures  in 
light  of  requirements  for  information  support.  A  scenario 
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of  information  use  in  performing  tasks  is  developed. 

At  this  point,  we  are  still  dealing  conceptually  with 
"information  support  services"  rather  than  with  particular 
"information  systems".  This  is  true  because  the  user  may 
require  several  related  reports  produced  by  different  sys¬ 
tems  in  order  to  perform  effectively  a  function.  The  user 
may  in  fact  be  totally  ignorant  of  which  system  is  gener¬ 
ating  a  specific  output.  Analysis  should  cover  not  only 
the  current  use  of  information  services,  but  potential  use 
as  well. 


•  Selection: 


Characteristics 


Planning 
Management 
Communication 
Data  Base 
Monitoring 

Reference 

Scientific 


Relevant  and  necessary 

Relevant  and  necessary 

Often  relevant  and  advisable 

Often  relevant  and  advisable 

Relevant  and  necessary  for  opera¬ 
tional,  possibly  tactical 

Often  relevant  and  advisable 

Relevant  and  necessary 


Activity  Levels  —  Strategic,  tactical  and  opera¬ 
tional 


Methods  —  Open-ended  depth  interview;  gaming, 
Delphi  technique 

•  Conduct:  Existing  documentation  is  not  apt  to  be 
helpful  for  this  approach,  and  heavier  reliance  is  there¬ 
fore  placed  on  an  open-ended  depth  interview.  A  checklist 
of  questions  follows: 

(1)  What  specific  use(s)  are  made  of  information  sup¬ 
port  services  in  performing  each  functional  task? 

(2)  What  additional  use(s)  could  be  made  of  each  in¬ 
formation  service  in  performing  each  task  if  minor 
changes  were  made  to  the  service? 
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(3)  Are  there  uses  for  which  an  information  service 
was  intended,  but  for  which  the  service  is  unsuitable 
or  ineffective?  What  are  the  problems? 

(4)  Are  there  uses  for  which  the  information  service 
was  not  intended,  but  for  which  the  service  proves  use¬ 
ful  in  performing  assigned  tasks?  What  procedural 
changes,  if  any,  were  made  to  take  unforeseen  advantage 
of  the  service? 

(5)  What  specific  effects  would  failure  of  a  support¬ 
ing  information  service  have  on  each  task?  Are  there 
backup  sources  of  information?  In  what  ways  are  these 
backup  sources  inferior  to  the  primary  information 
service? 

(6)  What  options  are  available  in  performing  assigned 
tasks?  What  are  the  possible  outcomes  of  each  task? 

(7)  What  alternative  courses  of  action  are  influenced 
by  available  information  support? 

During  analysis  of  interviews,  the  evaluation  team  is 
apt  to  discover  differences  of  opinion  concerning  the  de¬ 
sired  or  actual  use  of  information  support  services.  Such 
differences  may  occur  because  of  the  diversity  of  functions 
performed  by  user  groups  or  because  of  the  va lying  breadth 
of  perspective  associated  with  each  management  decision¬ 
making  level. 

Depending  on  the  criticality  of  the  area  in  which  a  dif¬ 
ference  of  opinion  might  occur  and  on  the  degree  of  polarity 
of  user  views,  the  evaluation  team  may  decide  to  utilize 
more  expensive  operational  gaming  technioues  and/or  Delphi 
techniques  to  attempt  resolution  of  subjective  opinions. 
Gaming  techniques  are  generally  more  appropriate  to  assess 
personnel  reactions  and  behavior  under  simulated  situations. 
Approaches  assist  in  more  consistent  description  of  pre¬ 
scribed  procedures  for  performing  tasks. 

•  Result:  At  the  end  of  this  step,  an  overall  func¬ 
tional  diagram  of  the  organization  should  be  developed 
which  depicts  major  functions  (primarily  physical  processes 
such  as  production)  and  major  information  networks.  The 
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diagram  should  be  accompanied  by  a  written  scenario  of 
information  use  by  each  functional  group.  Both  the  diagram 
and  the  scenario  description  serve  as  a  reference  point  for 
subsequent  evaluation  steps  which  reexamine  information 
service  usage  at  a  finer  level  of  detail. 

c.  Forms  of  Data  and  Interactions  Involved 

•  Purpose:  To  examine  the  nature  of  data  supplied  by 
and  provided  to  each  function  to  support  defined  tasks. 
Interactions  between  groups  in  supplying  and  using  informa¬ 
tion  are  an  important  part  of  the  description.  The  objec¬ 
tive  of  this  analysis  is  to  be  able  to  trace  the  flow  of 
each  data  element  from  its  source  to  its  ultimate  use(s) 

in  performing  functional  tasks. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitoring 

Scientific 


Sometimes  relevant 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Sometimes  relevant 
Relevant  and  necessary 
Relevant  and  advisable 


Levels  —  Strategic,  tactical,  operational,  execution 


Method  —  Observation,  interview,  analysis  of  forms 

•  Conduct:  Samples  of  the  paperwork  involved  in  col¬ 
lecting,  processing  and  interpreting  data  should  serve  as 
the  basis  for  supplementary  observation  or  structured 
interviews.  Examples  of  such  paperwork  are  data  prepara¬ 
tion  forms,  computer  printouts,  hard-copy  of  cn-line  trans¬ 
actions,  input  media  and  report  formats. 
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If  non-paper  media  (CRT,  voice  etc.)  are  used  for  col¬ 
lection,  and  retrieval  of  data,'  then  samples  should  be 
transferred  to  a  hard  copy  medium  (picture  of  CRT  or  script 
reduction  of  voice).  This  will  facilitate  planning  of  the 
interviews  to  assess  all  such  materials  jointly  by  the 
evaluation  team  and  affected  managers.  As  a  minimum,  mem- 

I 

bers  Of  the  evaluation  team  should  observe  data  collection 
and  retrieval  procedures  first  hand  in  order  to  prepare 
notes  as  the  basis  for  interviews. 

In  the  process  of  analyzing  t|ie  forms,  printouts,  etc. 
in  detail  and  conducting  interviews,  the  following  should 
be  considered: 

I 

(1)  Describe  identify  briefly  the  data  in  functional 
terms . 

i 

(2)  Identify  groups  required  to  collect,  analyze, 
search,,  interpret  and/or  summarize  data  for  functions 
being  reviewed . 

(3)  Identify  groups  which  request  the  group  under 
study  to  collect,  analyze,  search,  interpret  or  sum¬ 
marize  the  data  for  their  use. 

(4)  1  Describe  groups  which  interact  during  effective 
use  of  data. 

Questions  2  and  3  serve  a  valuable  purpose  in  showing 
interrelationships  among  different  groups.  Omissions,  dis¬ 
crepancies  and  conflicts  can  generally  be  resolved  by 
tracing  data  flow  from  functional  tasks  back  to  the  source. 

•  Result:  The  tangible  output  of  this  activity  is  a 
set  of  matrices  depicting  the  flow  of  data  from  the  source, 
through  intermediate  levels  of  activity,  to  the  business 
function(s)  served.  Such  matrix  models  facilitate  analysis 
of  the  alternate  paths  of  data  from  source  to  functions. 
Specifically,  the  matrix  may  reveal  (a)  data  which  is  being 
collected  but  not  used  or  (b)  data  which  can  arrive  at  a 
business  function  through  several,  possibly  redundant, 
paths . 


3.  Identify  Alternative  Courses  of  Action*  For  decision¬ 
making  and  planning  activity  support,  it  will  he  necessary  to 
identify  the  major  decisions  to  be  made  and  to  describe  pos¬ 
sible  alternative  courses  of  action  in  terms  of  the  functional 
operations  served.  Then  the  information  required  to  make  a 
choice  among  alternatives  must  be  analyzed.  Study  of  the  de¬ 
cisions  involved  must  assess  for  each  alternative: 

a.  Payoffs  and  consequences  if  selected. 

b.  Risk  and  utility  of  related  course  of  action. 

c.  Probabilities  of  selection  and  success  thereafter. 

This  is  a  particularly  difficult  and  time  consuming  step. 

It  involves  technical  preparation  in  decision  analysis  and 
skill  in  interpreting  empirical  utility  estimates  on  the  part 
of  the  evaluation  team.  The  system  user  must  also  exhibit 
substantial  patience  and  cooperation  to  derive  full  under¬ 
standing  and  benefit. 

a.  Description  of  Alternatives  in  Funr  ^onal  Terms 

•  Purpose:  To  identify  the  kinds  of  decisions  made 
and  activities  controlled  as  part  of  task  performance.  Foi 
each  such  decision  or  activity,  the  available  alternatives 
are  spelled  out  and  described  in  operational  teims. 

•  Selection: 


Characteristics 


Planning 

Management 

Data  Base 

Communication 

Monitoring 

Reference 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  advisable 
Rarely  applicable 
Relevant  and  necessary 
Relevant  and  advisable 
Sometimes  relevant 


Activity  Levels:  Strategic,  tactical  and  opera¬ 
tional 
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Method : 


Interview,  analysis  of  documentation 

•  Conduct:  The  interview  should  be  conducted  within 
the  context  of  the  just  developed  scenario  of  information 
use.  Questions  6  and  7  of  the  previous  step  are  especially 
pertinent  and  should  serve  as  the  basis  for  this  interview. 
Documentation  may  describe  the  programmed  decisions  ade¬ 
quately  for  operational  systems.  In  this  phase,  the  study 
team  enumerates  a  comprehensive  list  of  the  viable  alter¬ 
natives  for  each  situation  involving  decisions  about  or 
control  over  execution  of  tasks.  This  should  be  done  in 
terminology  characteristic  to  the  function  under  study.  A 
checklist  of  questions  follows: 

(1)  What  are  the  major  kinds  of  decisions  made  or 
activities  controlled?  Relate  these  to  task  per¬ 
formance  . 

(2)  What  viable  options  and  alternatives  are  avail¬ 
able  for  each  decision  or  activity? 

(3)  What  possible  events  and  parameters  influence 
selection  of  an  option  or  alternative  course  of 
action? 

•  Result:  A  list  of  alternative  courses  of  action  for 
each  decision  or  point  of  control  related  to  specific  tasks 
Each  should  be  discussed  in  terms  of  the  variables  which 
influence  selection  of  the  alternative. 

b.  Information  Required 

•  Purpose:  To  describe  the  specific  information  re¬ 
quired  to  select  each  identified  alternative. 

•  Selection: 


Characteristics 


Planning 
Management 
Communication 
Data  Base 
Monitoring 


Relevant  and  necessary 
Relevant  and  necessary 
Normally  relevant 
Normally  relevant 
Normally  well-defined 
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Reference  Sometimes  relevant;  expensive 

Scientific  Sometimes  relevant 

Activity  Levels:  Strategic,  tactical,  operational 

Method:  Interview,  observation,  gaming 

•  Conduct:  The  interviews  snould  be  conducted  as  an 
item  by  item  review  of  alternatives.  Determination  of  all 
the  information  required  to  choose  an  alternative  may  cake 
considex’able  prodding  and  skillful  questioning  on  the  part 
of  the  interviewer.  Situational  gaming,  in  the  form  of 
posing  hypothetical  decision  situations  to  the  manager,  and 
observing  reactions,  is  often  useful  in  discovering  and 
verifying  what  information  is  actually  needed.  Actual  per¬ 
formance  can  also  be  observed  by  "in  basket"  or  activity 
sampling  techniques.  The  care  and  patience  which  must  be 
exercised  in  this  procedure  make  the  activity  time-consuming 
and  therefore  expensive. 

The  default  option,  appropriate  when  insufficient  in¬ 
formation  is  available,  is  an  important  part  of  this  anal¬ 
ysis.  The  default  option  takes  the  form  of  maintaining 
the  status  quo  or  postponing  positive  action  until  neces¬ 
sary  information  is  available.  Such  passive  alternatives 
should  be  considered  as  viable  options  for  the  decision 
along  with  all  positive  actions. 

Following  is  a  checklist  of  questions  to  guide  the 
interviewer: 

(1)  What  ir  formation  is  necessary  before  selecting 
each  alternative? 

(2)  'vhat  is  the  role  of  information  in  increasing  or 
decreasing  the  probability  of  success  for  each  alter¬ 
native? 

(3)  Would  more  or  less  detailed,  accurate,  tiinelv, 
relevant,  or  reliable  information  alter  your  decision? 
What  are  the  upper  and  lover  limits  for  each  factor 
which  would  leave  the  decision  unaltered? 
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(4)  At  what  point  is  decision  or  control  action 
taken  no  matter  what  additional  information  is  re¬ 
ceived? 


•  Result:  Facts  elicited  in  this  investigation  should 
be  summarized  in  tabular  form,  clearly  depicting  the  alter¬ 
natives  available  for  each  decision  and  the  information  re¬ 
quirements  for  each.  Diagrams  such  as  decision  trees  show¬ 
ing  sequences  of  decisions  and  all  possible  paths  for 
alternatives,  are  useful  outputs  at  this  stage.  Subsequent 
steps  cover  identification  and  further  definition  of  the 
branches  and  options  identified. 

c.  Goals  an.l  Consequences  of  Alternatives 

•  Purpose:  To  identify  all  possible  outcomes  of  each 
alternative  in  terms  of  both  the  effects  and  consequences 
that  may  result  if  the  alternative  is  selected  and  imple¬ 
mented  . 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Bsse 

Moni to ring 

Reference 

Scientific 


Relevant  and  necessary 

Relevant  and  necessary 

Rarely  applicable 

Sometimes  relevant 

Desirable  for  constrained  area 

Rarely  applicable;  expensive 

Sometimes  relevant  and  advisable 


Activity  Levels:  Strategic,  tactical  and  opera¬ 
tional 

Method:  Interview,  Delphi  techniques 

•  Conduct:  The  interview  in  this  phase  builds  upon 
and  amplifies  the  results  of  eho  previous  two  steps. 
Again,  this  step  is  time-consuming  and  patience  is  needed 
to  get  accurate  arid  comprehensive  results.  Delphi  may  be 
required  to  predict  possible  consequences  of  important 
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decisions  for  which  the  organization  has  little  or  no 
precedent  or  where  subjective  estimates  vary  widely.  Tac¬ 
tical  managers  within  the  organization  and,  possibly,  out¬ 
side  experts  should  oe  included  in  such  a  Delphi  study. 

Following  is  a  checklist  of  questions  to  guide  the 
evaluator  during  the  interview.  These  same  questions  can 
serve  as  the  basis  for  a  Delphi  study,  however  they  should 
be  directed  to  specific  issues  and  situations  for  this  pur¬ 
pose. 

(1)  What  goal(s)  is  associated  with  each  alternative 
under  consideration? 

(2)  What  other  effects  or  consequences  may  result  if 
each  alternative  is  attained?  How  are  these  possibil¬ 
ities  affected  by  events  over  which  you  have  no  con¬ 
trol? 

(5)  What  is  the  source  of  determining  the  likely  ouc- 
oome(s)  of  applying  an  alternative?  (If  subjective,  a 
Delphi  study  is  suggested.) 

•  Result:  The  findings  of  this  phase  should  be  in¬ 
corporated  with  the  results  of  the  previous  steps  into  the 
decision  tree  diagi'ams  which  were  roughly  stated  in  step 
A3b.  They  should  indicate  all  feasible  outcomes  from  se¬ 
lection  of  each  possible  course  of  action.  The  decision 
tree  should  indicate  the  sequence  of  decisions  involved  in 
performing  a  task.  Branch  points  or  nodes  should  clearly 
differentiate  management  alternatives  from  chance  or  ex¬ 
ternal  influences. 
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B.  Assess  the  Current  State  of  the  Information  System 


1 .  Define  the  System  Characteristics 

The  goals  and  functions  of  each  system  affecting  an  operation¬ 
al  activity  area  should  be  reviewed,  noting  carefully  the  changes 
in  purpose  or  benefits  derived  in  comparison  to  earlier  documenta¬ 
tion.  A  scenario  should  be  developed  to  describe  activities  of 
various  groups  interacting  with  and  using  the  system.  Paramecers 
to  measure  the  system  effects  on  its  users  and  criteria  of  satis¬ 
factory  performance  should  be  developed.  This  may  again  entail  the 
transformation  of  subjective  opinions  into  tangible  measures. 


The  degree  to  which  expected  benefits  are  realized  will  depend 
greatly  on  the  user  role  in  development,  project  team  discipline, 
and  realistic  planning.  The  evaluation  team  should  not  be  surprised 
that  serendipity  often  provides  unintended  benefits  or  shifts  in 
utilization  which  drastically  alter  the  worth  or  objectives  of  an 
existing  system. 

As  an  important  adjunct  to  this  evaluation  phase,  the  adequacy 
of  computer  system  performance  should  be  assessed  in  light  of  the 
enhancement  or  detraction  from  attaining  system  goals.  The  com¬ 
puting  environment  including  standards,  controls,  security,  and 
personnel  .qualifications,  is  also  a  strong  influence. 

a .  System  Goals  and  Functions  Served 

•  Purpose:  To  review  system  goals  and  relate  the  sys¬ 
tem  to  the  organizational  functions  served.  Emphasis  is 
placed  on  distinguishing  loth  intended  goals  and  the  actual 
current  use  to  which  the  system  is  being  put. 

•  Selection: 


Characteristics 


Planning 
Management 
Communication 
Data  Base 


Relevant  and 
Relevant  and 
Relevant  end 
Relevant  and 


necessary 

necessary 

necessary 

necessary 
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Reference 

Monitor- 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 


Activity  Levels  —  Strategic,  tactical,  operational 
and  execution 


Methods  —  System  documentation,  interviews 


•  Conduct:  Existing  system  documentation  should  be 
examined  to  determine  tne  stated  goals  and  the  functions 
each  was  intended  to  serve.  The  evaluation  team  should  as 
sess  the  accuracy  and  currency  of  system  documentation. 
Stated  goals  are  very  likely  to  have  been  supplanted  by 
new  ones,  either  formally  or  informally. 

Information  support  services  should  be  reviewed  in 
detail  (part  A2)  and  the  role  of  specific  systems  in  pro¬ 
viding  each  service  should  be  clearly  delineated.  The 
interdependencies  among  systems  necessary  to  provide  ef¬ 
fective  service  should  also  be  noted  at  this  point. 


The  interview  is  a  useful  technique  for  assessing 
the  changes  in  purpose  of  or  benefits  expected  from  a  sys¬ 
tem  in  comparison  to  earlier  documentation.  Following  is 
a  checklist  of  questions  to  guide  the  interviewer: 


(1)  What  information  system(s)  provide  necessary 
support  services? 


(2)  '-.lint  is  the  current  status  of  each  system  and 
how  dees  this  af'ect  your  assessment? 

(a  )  Newly  ins  called? 

(b)  Fully  operational?  For  how  long? 

(c)  Under  revision? 

(3)  '..hat  technical,  economic  and  operational  benefits 


(a)  Were  expected  from  the  system? 

(h)  Have  been  realize:  by  the  system? 

(c)  Are  still  anticipate!  for  the  system? 


(4)  '.-/hat  problems  limit  the  effectiveness  of  each 

syscem? 
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(5)  l /hat  is  the  expected  useful  life  of  the  system? 

(a)  Could  system  modifications  extend  or 

shrink  the  useful  life?  Specify. 

(b)  Could  environmental  changes  extend  or 

shi’ink  the  useful  life?  Specify. 

(6)  What  growth  in  emphasis,  use,  transactions,  and/ 

or  files  can  be  anticipated? 

•  Result:  This  phase  of  the  analysis  should  result  in 
a  basic  understanding  of  the  relationship  between  functional 
requirements  and  information  systems.  All  change?  in  func¬ 
tional  re  luirements,  evolution  in  use,  and  modifications  to 
systems  should  be  explicitly  documented.  Finally,  systems 
should  be  assessed  according  to  their  ability  to  fulfill 
current  and  future  needs  for  information  support. 

b.  Scenario  of  System  Use 

•  Purpose:  To  develop  a  written  description  of  actual 
interaction  among  user  groups  and  the  system.  This  will 
include  operational  description  of  system  use,  effects  of 
the  system  on  users,  satisfaction  of  users,  and  impact  of 
the  output  on  decisions  or  control  of  activity. 

•  Selection 


Characteristics 


Pla  nn:ing 

Management 

Communication 

data  B-se 

Reference 

Monitor 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Relevant,  usually  simple  pattern 
Relevant  and  necessary 
Relevant  and  necessary 
Usually  implicit  in  documentation 
Relevant  and  desirable 


Activity  Levels  —  Strategic,  tactical,  operational, 
execution 

Method  --  Unstructured  interview;  observation;  gamin j 
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•  Conduct:  The  scene rio  description  of  information 
system  usage  should  be  obtained  primarily  through  unstruc¬ 
tured  interviews  of  system  users  combined  with  direct  ob¬ 
servation  of  system  operation. 

Results  of  the  previous  phase,  matching  systems  with 
the  information  support  services  they  provide,  should  be  in 
hand  during  development  of  the  scenario  of  information  sys¬ 
tem  usage.  This  will  help  to  insure  that  variations  in 
system  usage  will  be  included  in  the  scenario  description. 
Situational  descriptions  can  be  used  to  supplement  knowledge 
of  procedures  by  using  operational  gaming  techniques.  Re¬ 
actions  to  these  situations  can  be  more  authentic  when  pre¬ 
sented'  in  an  impromptu  fashion.  In-basket  analysis  can 
give  a  picture  of  actual  data  flows  and  can  serve  as  the 
basis  for  observation  of  system  users  under  controlled, 
simulated  conditions. 

Following  is  a  checklist  of  questions  to  guide  the 
interviewer  and  to  direct  attention  to  important  observa¬ 
tions  during  simulation  of  system  usage. 

(1)  Whet  is  tne  nature  of  interaction  with  the  system? 

(a)  What  means  of  access  cr  communication? 

(b)  Describe  Forms  of  input  and  output? 

(2)  When  to  you  use  each  system? 

(a.)  How  frequently? 

(b)  Under  what  conditions? 

(c)  What  are  the  restrictions  on  use? 

(3)  How  arc  basic  functions  performed? 

(a)  Describe  procedures  for  creating,  modifying 

and  aercssinj  files. 

(b)  Describe  responsibilities  for  error  coirection 

and  verification  of  data  inputs. 

(c)  Describe  ei forts  to  interpret  and  analyze  sys¬ 
tem  oueputs. 

(4)  What  is  the  purpose  of  the  system? 

Contrast  various  views  of  purpose  among  functional 

users  with  the  foi'mally  stated  objectives  and  goals. 


Ill -2  3. 


Differentiate  the  goe^s  at  various  levels  of  manage¬ 
ment  and  activity. 

•  Result:  This  phase  of  the  analysis  should  result 
in  a  written  description  of  actual  system  usage  in  scenario 
form.  It  should  include  all  significant  variations  of  ex¬ 
pected  usage  which  provide  different  information  support 
services.  This  scenario  is  contrasted  with  system  docu¬ 
mentation  to  produce  a  summary  of  variances  between  current 
and  intendc J  system  function. 
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e.  System  Evaluation  Measures  and  Criteria 

*  Purpose:  To  define  quantitative  measures  and 
criteria  useful  for  system  evaluation.  The  criteria  should 
indicate  threshold  levels  of  satisfactory  system  perfor¬ 
mance.  Subjective  opinions  must  be  translated  into  tangible 
measures  of  effectiveness.  Preferably,  this  step  should  in¬ 
volve  only  updating  of  measures  identified  during  system 
design. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Normally 
Relevant 
Possibl y 
Relevant 
Relevant 
Relevant 
Possibly 


relevant  and  advisable 

and  necessary 

relevant 

and  desirable 

and  desirable 

and  necessary 

relevant 


Activity  Levels  —  Strategic,  tactical,  operational 
and  execution 


Methods  —  Unstructured  interview;  Delphi;  analysis 
and  gaming  based  on  scenario;  accounting  and  performance 
records . 

•  Conduct:  The  evaluation  team  should  start  this 
program  phase  by  reviewing  system  goals  and  proposed  meth¬ 
ods  of  measuring  in  order  to  assess  these  in  light  of  the 
functions  served  and  information  support  required.  Gen¬ 
eral  goals  must  he  refined  to  specific  objectives  which 
make  clear  what  constitutes  satisfactory  system  perfor¬ 
mance.  Typically,  tliis  involves  identifyirg  critical 
pararnecers  ol  the  system  and  speed  lying  the  levels  to  be 
attained  for  satisfactory  performance.  This  step  is  ex¬ 
tremely  difficult  and  time-consuming  if  objectives  and 
measurement  of  progress  were  not  articulated  in  the  ini¬ 
tial  design. 
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A  measure  of  effectiveness  (MOE)  is  generally  an 
operational  factor  which  is  expected  to  vary  directly  with 
information  system(s)  effects.  Functional  performance 
ratios  are  frequently  good  candidates  for  such  measures. 

For  example,  a  production  control  system  might  use  the  pro¬ 
portion  of  indirect  to  direct  employees  as  an  MOE.  Of  real 
interest  is  the  percent  or  dollar  value  of  improvement 
which  can  be  attributed  to  the  information  system.  In  order 
to  isolate  system  effects,  it  will  be  necessary  to  review 
historical  data  on  operations  under  varied  conditions  and 
processing  methods.  Where  possible,  the  factors  analyzed 
should  be  also  used  for  operational  purposes  in  order  to 
reduce  bias  in  the  statement  of  benefits  derived  from  the 
system. 

User  opinions  of  the  technical,  economic  and  opera¬ 
tional  benefits  expected  from  a  system,  developed  earlier, 
are  valuable  inputs  in  the  formulation  of  a  MOE.  In  the 
absence  of  well  defined  measures,  general  concensus  among 
users  concerning  an  expected  benefit  should  be  emphasized 
in  formulation  of  a  MOE.  Normally,  careful  delineation  of 
an  expected  benefit  will  make  evident  a  corresponding  MOE. 

If  a  wide  disparity  of  opinions  exists  among  users  concern¬ 
ing  expected  benefits,  the  Delphi  technique  may  prove  to  be 
useful  for  reaching  concensus.  The  Delphi  technique  may 
also  prove  useful  in  ordering  and  compacting  an  initial  set 
of  possible  MOEs>.  deveral  members  of  such  a  set  may  cor¬ 
relate  so  highly  that  only  one  of  the  measures  need  be  re¬ 
tained.  Carr*  must  be  taken  co  differentiate  the  desired 
bene'fts  at  various  levels  of  management  and/or  activity. 

In  some  cases,  historical  data  can  be  used  to  perform 
a  regression  analysis  in  order  to  determine  the  contribution 
of  various  factors  to  system  success.  Often  the  operational 
parameters  used  to  measure  improved  performance  relate,  to 
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normally  kept  statistics.  In  many  cases,  tho  parameters 
can  be  expressed  in  terms  of  dollars  by  using  accounting 
practice  already  established. 


Structure  is  given  to  the  interview  through  use  of  a 
predetermined  list  of  general  performance  factors.  The 
list  includes  factors  most  frequently  significant  in  a 
variety  of  functions,  and  the  user  may  choose  to  define  ad¬ 
ditional  ones  of  unique  concern  in  a  particular  application. 
This  set  is  useful  in  comparing  views  of  a  system  from 
varied  sources  and  in  comparing  different  systems. 

For  each  factor,  the  user  is  asked  to  rate  his  degree, 
of  concern  in  using  the  system  over  impact  on  operations, 
and  actual  or  anticipated  performance  by  the  system. 

Ratings  are  subjectively  weighted  at  one  of  five  levels 
a  s  foil  ov/s : 


We  3  glit 


Degree  of  Concern 


4 


1 

0 


Critical  or  major 

Cpecial  attention  necessary 

Desirable,  standard  approaches  adequate 

Hot  considered 

Insignificant 


Weight 

4 

2 

1 

0 


Actual  or  Anticipated  Performance 
Notably  effective 

b-'tisfactovy;  improvement  possible 
Effects  not  i  Lscernable 
Annoying;  improvement  desirable 

Detrimental;  correction  of  serious  flaws 

man  intory 


Following  J?  tie  list  ol  general  performance  factors 
to  be  vat*  ’  in  Table  III-l. 
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TABLE  HI-l 

. _  ' PERFORMANCE  CONSIDERATIONS 

i  Concern 

Weight 

J 

1.  Gathering  and  recording  data 

2.  Accuracy  ani  verification  of  input 

3k  Editing  and  encoding  of  data 

4.  Transaction  flow  characteristics  and 
growth  potential  ■ 

b.  Mutual  dependence  of  users  'and  files 
b."  Pile  size  and  growth  potential 
1 .  Retention  and  intejrity  of  files 
8f  Compatibility  of  data  among  systems 
9.  Duplication  of  effort  or  records 

10.  Response  time  , 

11.  Interaction  or  intervention  during 

processing  i 

12.  .  Accessibility  end.  ease  of  use  , 

13.  Byproduct  use'  of  data 

14.  Format  variability  in  output  i 

15.  Reproducible  results  ;  : 

16.  Graphics 

17.  Output  retention  ani  distribution  ■ 

18.  Communi cation  among  user  locations 

19.  Comparison  among  trials 

]  , 

20.  Statist ical  ma lyses  i 

21.  Content  retrieval 

I 

22.  Skill  of  user  nn  programming  , 

23.  Independence  of  hardware/software 

24.  Control  over  operation  of  system 

25.  Flexibility,  in  Changing  parameters 

or  proce  lures  : 

26.  Manual  effort  reduction 

27.  Training  required 

28.  Analysis  of  system  activity 

29.  Reliability  am  backup 

- i-  -  i 

30.  1  Schedule  restribtions 

■  i 


Performance 

Weight 
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•  Result:  This  phase  of  the  analysis  should  provide 
a  set  of  measures  of  effectiveness  which  reflect  the  true 
objectives  of  the  system.  Some  effort  should  be  ma  Je  to 
minimize  this  set  by  eliminating  redundant  MOEs.  A  level 
of  achievement  should  be  established  for  each  HOE  which 
is  the  minimum  required  to  achieve  "satisfactory"  system 
performance.  Also,  a  compensatory  objective  function  may 
be  established  to  relate  the  various  parameters. 

A  description  of  chosen  MOE,  methods  for  measurement 
and  analysis,  and  factors  influencing  the  choice  of  a  MOE 
should  be  included  in  a  formal  report.  This  report  should 
be  reviewed  and  explicitly  approved  by  management  author¬ 
izing  the  evaluation  study. 

d.  Computer  System  Performance 

•  Purpose:  To  assess  the  role  of  computer  performance 
faccors  in  influencing  the  effectiveness  of  the  system. 

Each  performance  factor  must  be  considered  relative  to  its 
potential  impact  on  operations.  For  this  reason,  the  degree 
of  operational  concern  over  each  factor  must  be  assessed  as 
well  as  actual  computer  system  performance  for  each. 

«  Selection: 


Characteristics 


Planning 

Ma  na  .jement 

Communication 

Data  Bast1 

Reference 

Monitor 

Scientific 


Sometimes  relevant 
Sometimes  relevant 
Relevant  and  necessary 
Relevant  and  necessary 
Often  relevant 
Relevant  and  necessary 
Often  relevant  and  advisable 


Activity  Levels  --  Operational,  execution 

Methods  —  Structured  interview,  analysis  of  system 
and  department  documentation. 
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•  Conduct;  At  this  phase,  operational  limitations 
or  characteristics  caused  by  the  computer  system  are 
analyzed.  Thus,  the  environment  in  which  the  information 
system  must  work  is  characterized  and  contrasted  with  ex¬ 
pected  or  desired  situations.  This  normally  consists  of 
reviewing  and  updating  department  and  system  documentation. 

In  addition  to  subjective  estimation  of  performance 
factors,  actual  measurement  of  certain  highly  quantitative 
factoi’S  is  often  feasible  at  very  little  expense.  In  many 
cases,  the  system  itself  will  have  facilities  for  expedit¬ 
ing  its  own  performance  measurement.  Performance  factors 
like  response  time,  file  size  and  growth,  and  reliability 
are  exceptionally  good  candidates  for  this  sort  of  treat¬ 
ment. 

The  particular  items  related  to  computer  system  per¬ 
formance  whicn  should  be  recorded  are  outlined  in  Appendix 
C.  Emphasis  should  be  on  estimates  and  existing  data 
related  to  use  and  operation  of  the  information  system. 

•  Result:  A  summary  of  computer  performance  findings 
which  limit  system  effectiveness  should  be  included  in  a 
formal  report;  perhaps  the  same  report  as  discussed  in  the 
previous  section.  System  Evaluation  Measures  and  Criteria. 
This  report  should  highlight  the  notably  effective  per¬ 
formance  'actors  ii.  the  system  as  well  as  problem  areas. 
System  problems  should  be  discussed  in  relation  to  their 
iegrnb ' n't  effects  on  performance  in  operational  areas. 
Sug.jOrtiTi  computer  system  changes  and  their  anticipated 
bene; its  should  also  be  reported.  Details  identified  in 
this  area  should  be  used  to  update  existing  documentation 
about  organization  and  computer  system  utilization.  The 
impressions  onj  feelings  about  the  system  will  be  verified 
later  in  the  phase  related  to  actual  measjrement  of  ac- 

t  Lvit''. 
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2 .  Audit  Economics  of  the  Information  System 


Using  standard  job  and  task  definitions,  the  initial  in¬ 
vestment  in  the  system  is  assessed  and  then  reviewed  in  light 
of  revisions  implemented  and  planned.  Both  operating  expenses 
of  the  system,  including  programming  maintenance  and  user  in¬ 
volvement  time,  and  operating  savings  are  audited.  Where 
possible,  the  results  are  compared  to  prior  operations  or 
alternative  methods.  To  the  extent  accounting  procedures  al¬ 
low,  resultant  changes  in  performance  are  translated  to  vari¬ 
ances  in  expense  dollars. 

a .  Investment 

•  Purpose:  To  evaluate  the  fixed  development  cost  of 
each  system.  This  must  include  the  initial  investment  as 
well  as  all  subsequent  costs  incurred  for  refinements  and 
additions  to  the  system.  It  presumes  that  standard  job 
and  task  definitions  exist  in  the  organization  conducting 
the  study. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant  and 
Relevant  and 
Relevant  and 
Relevant  and 
Relevant  and 
Relevant  and 
Relevant  and 


necessary 

necessary 

necessary 

necessary 

necessary 

necessary 

necessary 


Activity  Levels  —  Strategic,  tactical,  operational, 
execution 


•  Method  --  Accenting  and  project  management  records 

•  Conduct:  Records  dating  back  to  the  initial  stages 
of  systems  analysis  and  design  should  be  collected  for  this 
phase  of  the  analysis.  Considering  both  data  processing 
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and  customer  or  user  commitments,  the  initial  investment 
in  the  system  should  be  calculated,  taking  explicit  ac¬ 
count  of  the  following  aspects  in  the  development  of  each 
system: 

(1)  Systems  Analysis 

Definition  of  system  objectives;  feasibility 
study;  user  requirements. 

(2)  Systems  Design 

Specifications  for  programs,  procedures,  file 
structures,  input  and  output  report  content,  algo¬ 
rithms  and  models;  flow  charts  and  decision  tables; 
programming  conventions  and  standards. 

(3)  Programs 

Coding  or  acquisition  of  compute  programs  in 
operational  status. 

(4)  Testing 

Verification  that  software  and  hardware  performs 
according  to  specifications  and  requirements. 

(5)  Conversion 

Operational  changeover  of  files  and  procedures. 

(6)  Documentation 

User  manuals,  program  descriptions. 

( 7 )  Ha rdwa  re  a  dditions 

New  hardware  necessary  to  support  the  system. 

(8)  Planning,  orientation  and  education 

Indoctrination  of  users  and  operators  in  ef¬ 
fective  system  use. 

(9)  Overhead 

Project  management,  administrative  support  and 
personnel  services. 

In  seme  cases,  it  may  be  necessary  to  distribute 
these  coasts  ove.'  several  systems.  For  example,  new 
hardware  or  a  newly  converted  common  file  may  serve 
several  systems  and  overb°ad  costs  must  be  prorated 
among  the  systems. 

Costs  incurred  in  these  areas  subsequent  to  initial 
system  development  shouLd  also  be  obtained  from  accounting 
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records.  Care  must  be  taken,  however,  to  account 
for  the  time  value  of  money  by  discounting  all  costs 
back  to  a  standard  time  period — generally  the  period 
during  which  the  original  system  development  activity 
was  begun.  The  interest  rate  to  be  used  for  this 
cash  discounting  is  a  matter  of  corporate  policy. 

Fixed  system  costs  can  then  be  spread  over  the  entire 
expected  life  of  the  system,  facilitating  consideration 
of  the  original  investment  with  operating  costs  and 
savings. 

•  Result:  This  phase  of  the  analysis  should  yield  a 
summary  statement  of  all  costs  incurred  in  the  investment 
in  each  system.  By  expressing  these  in  terms  of  well- 
defined  major  tasks  and  jobs,  it  is  possible  to  compare  the 
effectiveness  of  development  among  systems  and  to  plan  more 
effectively  for  future  development. 

b.  Operating  expenses 

•  Purpose:  To  evaluate  the  recurring  expenses  asso¬ 
ciated  with  system  operations.  All  expense  levels  should 
be  measured  for  a  standard  accounting  period. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 


Activity  Levels  —  Strategic,  tactical,  operational, 
execution 


•  Method  —  Accounting  and  functional  activity  records 

•  Conduct:  Financial  data  and  activity  reports  from 
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recent  accounting  periods  should  be  collected  for  this 
phase  of  the  analysis.  Using  standard  job  and  task  def¬ 
initions,  operating  expenses  for  the  period  should  be 
calculated  taking  explicit  account  of  the  following  areas: 

(1)  Personnel 

(a)  Operations 

Supervision,  clerical,  equipment  operators 

(b)  Maintenance  of  system 

Preventive  and  corrective  hardware  mainte¬ 
nance;  corrective  software  maintenance;  operating 
system  improvement. 

(c)  User  involvement 

Time  spent  by  user  in  preparing  and  editing 
inputs,  collecting  and  interpreting  output’s. 

(2)  Supplies 

Forms,  cards,  tapes,  paper,  etc. 

(3)  Equipment 

Hardware  rental  charges  or  amortized  cost. 

(4)  Software 

Cost  of  using  software,  generally  based  on  utiliza¬ 
tion  of  scarce  resources 

(5)  External  services 

Programming,  consulting,  communications,  time 
sharing  etc. 

(6)  Overhead 

Ongoing  costs  for  operations  management,  adminis¬ 
trative  support  and  personnel  services. 

The  a  Dove  expenses  should  be  sampled  in  several 
recent  accounting  periods,  noting  variations  due  to  en¬ 
vironmental  and  system  changes.  Additionally,  there  may 
be  variations  due  to  seasonal  factors  in  system  usage 
which  must  be  considered  in  the  sampling  plan.  Care 
must  be  taken  in  this  phase  of  the  analysis  to  distin¬ 
guish  normal  on-going  costs  from  intermittent  costs. 
Large  intermittent  costs  which  occur  by  chance  in  the 
accounting  periods  under  study  could  invalidate  the 
analysis  if  not  distributed  over  time  to  give  a  fair 
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picture.  Particular  attention  must  be  given  to  ap¬ 
portioning  costs  consistently  among  systems  for  items 
that  are  shared.  For  example,  costs  of  running  and 
maintaining  an  operating  system  must  be  rationally 
apportioned  among  systems  for  various  time  periods. 

•  Result:  A  report  on  actual  expenses  is  the  basis 
for  estimating  the  financial  worth  and  life  of  the  system 
in  question.  The  investment  in  the  system,  allocated  over 
the  system  life  earlier,  can  then  be  combined  with  oper¬ 
ating  expenses  to  form  a  total  cost  statement  for  each 
system.  Further  analysis  of  the  costs  involved  in  system 
operations  can  be  a  worthwhile  venture,  often  revealing 
which  areas  offer  the  greatest  potential  for  cost  reduc¬ 
tions  . 


c.  Operating  savings 

•  Purpose:  To  identify  the  improvements  in  cost  of 
or  benefits  from  system  operation.  The  savings  are  ex¬ 
pressed  in  terms  of  cost  reduction  for  similar  services 
or  of  improved  services  for  a  lesser  increment  of  cost. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Not  relevant 
Perhaps  relevant 
Relevant  and  necessary 
Relevant  and  desirable 
Difficult  to  define 
Relevant  and  necessary 
Relevant  and  advisable 


Activity  levels  —  Operational  and  execution,  often 
tactical,  rarely  strategic. 

Method  —  Accounting  and  functional  activity  records. 

•  Conduct:  Records  may  support  direct  comparison  of 
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alternative  ways  to  provide  similar  services  or  to  com¬ 
pare  performance  before  and  after  system  changes.  If  ac¬ 
counting  and  controller  offices  have  agreed  upon  the 
measures  of  effectiveness  derived  earlier,  it  should  be 
possible  to  ascribe  dollar  value  to  changes  in  these 
functional  parameters.  This  normally  entails  negotiation 
of  the  amount  of  savings  or  benefits  which  can  be  attri¬ 
buted  to  system  improvements  alone.  For  instance,  a  re¬ 
duction  in  personnel  turnover  or  of  inventory  level  may 
be  aided  by  better  information  service,  but  is  normally 
also  affected  by  other  variables. 

Again,  standard  definitions  of  tasks,  jobs,  or  costs 
and  consistent  application  of  procedures  is  necessary  for 
meaningful  cost/savings  comparison  for  a  system.  The  ef¬ 
fects  of  different  systems  cannot  be  compared  unless  sub¬ 
stantial  discipline  is  achieved  in  defining  and  recording 
operational  data  related  to  costs. 

•  Result:  For  those  systems  which  afford  assessment 
of  savings,  the  return  on  investment  in  the  system  can  be 
calculated  for  its  estimated  life.  Decisions  can  be  made 
about  continuing  operation,  redesign,  or  replacement  of 
the  system. 
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3.  Measure  Information  System  Activity »  In  this  phase,  all 
tangible  evaluation  factors  should  be  measured  to  assess  actual 
versus  expected  performance.  This  would  involve  comparison  of 
ratios  related  to  expected  resource  use  and  benefit  achievement 
over  several  time  periods.  Analysis  of  activity  should  include 
observations  as  well  as  statistics  recorded  by  the  system  it¬ 
self.  Accuracy  of  input  should  be  verified  and  related  to 
earlier  subjective  views  of  confidence  in  output.  Particular 
attention  should  be  given  to  determining  the  sensitivity  of 
system  output  to  changes  in  input  from  either  the  function 
served  or  the  environment  of  the  system.  Statistics  should  be 
analyzed  on  various  aspects  of  system  use  and  growth  to  verify 
both  projections  and  subjective  opinion.  A  single  system  must 
be  placed  in  context  with  related  systems  and  functions  served. 
The  growing  trend  to  broaden  perspective  and  to  integrate  more 
areas  suggests  that  a  system  will  normally  impact  others,  either 
with  demands  and  side  effects  or  with  added  benefits  recognized 
elsewhere.  If  so,  the  audit  should  include  those  credits  or 
debits  for  the  system  being  evaluated. 

a.  Accuracy  and  Control  over  Input 

•  Purpose:  To  assess  accuracy  of  input  data  and  ef¬ 
fectiveness  of  controls  over  data  gathering,  transcription 
or  encoding,  transmission,  and  editing.  A  description  of 
the  linkages  between  data  sources  and  reports  or  listings 
for  various  functions  will  also  be  developed.  Included  are 
counts  of  transaction  or  message  activity  and  structure  or 
format  of  the  different  types  of  input. 

•  Selection: 

Characteristics 

Planning  Relevant  to  consistent  and 

representative  samples  over 
time 

Managempnt  Relevant,  but  may  have  trade¬ 

offs  with  timeliness  and/or 
cost 
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Communication  Relevant  and  necessary 


Data  Base 


Reference 

Monitor 

Scientific 


Relevant  and  necessary  for 
operational,  possible 
tactical 

Relevant;  usually  a  problem 
of  completeness 

Relevant  and  necessary 

Relevant  and  advisable 


Activity  Levels  —  Tactical,  operational,  and 
execution 


Methods  —  Work  sampling,  procedure  analysis,  matrix 
or  graphical  description;  hardware  and/or  software  moni¬ 
tors. 


*  Conduct:  Work  measurement  methods  can  be  used  to 
determine  the  amount  of  effort  expended  by  users  in  pre¬ 
paring  inputs,  verifying  entries,  and  making  corrections. 
Work  sampling  procedures  can  be  used  extensively  to  deter¬ 
mine  the  volume  and  nature  of  input,  including  proportion 
by  type  and  frequency  of  errors.  Traditional  statistical 
methods  can  be  used  to  determine  factors  such  as  required 
sample  size  or  confidence  limits.  Integrated  Procedures 
Control  (IPC),  outlined  by  Richardson  in  reference  38  of 
chapter  II,  provides  methods  for  systematic  collection  of 
pertinent  facts  regarding  inputs,  outputs,  files  and 
transactions  involving  these.  Use  of  IPC  facilitates  sub¬ 
sequent  analysis  of  data  forms  by  standardizing  data  collec¬ 
tion  procedures  and  allowing  summaries  of  facts  regarding 
data  forms  to  be  prepared  in  a  variety  of  ways. 

Input/output  charts  provide  a  visual  portrayal  of  sys¬ 
tem  inputs  and  their  relationships  to  reports  ov  input  to 
other  systems.  Expansions  of  this  concept  show  the  rela¬ 
tionships  between  data  elements  at  source,  in  messages,  in 
files,  and  in  outputs  while  indicating  frequency  of  use  and 
both  the  size  and  means  of  transmission.  Matrix  models  de¬ 
picting  data  paths  from  input  sources  to  output  functions 
can  be  used  to  study  the  flow  of  data  through  the  system. 
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This  analysis  may  reveal  alternative  paths  which  can  be 
used  to  audit  or  validate  transactions  within  the  system 
relative  to  inputs.  Alternatively,  analysis  may  reveal 
redundant  processing,  in  which  case  certain  input  sources 
may  be  eliminated.  The  relationships  among  activities  and 
data  flow  may  also  be  shown  by  graphic  methods.  (See 
Table  II-l.) 

•  Result:  The  volumes  and  descriptions  of  all  types 
of  inputs  are  recorded  and  the  nature  and  frequency  of 
errors,  including  omitted  and  faulty  transactions,  are 
indicated.  This  can  then  serve  as  the  basis  for  a  data 
flow  analysis  which  shows  the  effectiveness  of  gathering 
and  controlling  input  and  the  relationship  to  output. 

This  phase  results  in  a  formal  representation  of  data 
flows  and  estimates  of  the  accuracy,  cost,  and  timeliness 
of  data  available  to  various  functions  or  other  systems. 

b.  Performance  Versus  Expected  Benefits 

•  Purpose:  To  determine  the  accomplishment  of  the 
system  with  regard  to  the  criteria  and  measures  of  effec¬ 
tiveness  established  earlier.  The  effects  of  each  system 
are  assessed  to  develop  both  qualitative  and  quantitative 
estimates  of  the  level  of  attainment  of  each  M.O.E. 

•  Selection: 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant,  but  effects  may  be 
ill -defined 

Relevant  and  necessary 
Relevant  and  desirable 
Relevant  and  desirable 
Relevant  and  desirable 
Relevant  and  necessary 
Relevant  and  desirable 


Activity  Levels  —  Strategic,  tactical,  operational, 
execution 
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Methods  —  Observation;  accounting  and  operational 
records;  structured  interview 

•  Conduct:  The  very  nature  of  each  MOE  will  limit 
the  selection  of  measuring  methods  and  the  evaluation 
team  will  need  to  carefully  weigh  the  cost  of  measuring 

to  determine  appropriate  methods.  Least  cost  is  associated 
with  results  generated  as  by-products  of  normal  operation. 
Records  already  kept  by  the  organization  often  contain  the 
information  needed  for  MOE  calculations.  In  many  in¬ 
stances,  these  are  records  generated  and  maintained  by  the 
information  system  itself.  Again,  the  use  of  statistical 
analysis  allows  control  over  the  accuracy  and  precision  of 
the  measurement  process. 

Occasionally,  the  evaluation  team  will  need  to  ob¬ 
serve  operations  first-hand  in  order  to  evaluate  measures 
of  effectiveness.  Thi  a  be  a  time-consuming  and  costly 
project.  If  such  costs  are  high,  and  no  alternative  means 
of  measurement  are  feasible,  then  the  value  of  the  MOE 
should  be  reconsidered.  A  related  parameter  may  be  found 
which  is  more  amenable  to  measurement.  Hardware  and  soft¬ 
ware  monitoring  and  measuring  devices  may  be  useful  at  this 
stage  for  systems  geared  to  the  operational  control  and 
functional  execution  activity  levels. 

Structured  interviews  may  be  used  to  rate  the  merit 
of  the  system  in  relation  to  specific  functional  parameters 
or  objectives.  This  approach  is  necessitated  when  objec¬ 
tives  or  MOEs  are  ill-defined,  or  not  quantifiable,  and 
when  measurement  is  excessively  expensive,  difficult  or 
erratic. 

•  Result:  The  level  of  attainment  reached  for  each 
expected  benefit  should  be  compared  over  different  time 
periods  foi  trends  and  against  criteria  for  achievement 
success.  The  report  should  also  describe  any  special  con¬ 
ditions  or  extenuating  circumstances  which  make  the 


III-38 


measurement  results  suspect,  including  accuracy  or  pre¬ 
cision  of  measurements  and  subjective  bias.  Finally,  the 
report  should  contain  a  summary  view  of  system  success 
relative  to  critical  parameters  and/or  a  net  judgment 
based  on  weighted  qualitative  and  quantitative  factors. 

c.  Actual  Versus  Expected  Resource  Requirements 

Purpose:  To  compare  actual  resource  requirements 
with  those  expected  and  defined  during  planning  and  de¬ 
velopment.  Much  of  this  analysis  follows  from  economic 
analysis  of  investment  and  operating  costs  in  relation  to 
those  budgeted  or  advertised.  In  addition  to  the  cost 
items  mentioned  earlier,  special  attention  must  be  given 
to  computer  capacity  or  time  utilized  and  to  personnel 
skill  levels  needed.  File  storage,  software,  communica¬ 
tions,  and  personnel  effort  are  frequent  offenders  at  de¬ 
manding  more  capability  than  expected. 

Selection 

Characteristics 

Planning 
Management 
Communication 
Data  Base 
Reference 
Monitor 
Scientific 

Activity  Levels  —  Strategic,  tactical,  operational 
and  execution 

Y  :hods  —  Observation;  project  management,  opera¬ 
tional  and  accounting  records;  feasibility  studies, 
project  authorizations  and  budgets;  hardware  and  soft¬ 
ware  monitors. 


Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
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•  Conduct;  Ratios  should  be  developed  for  all  kinds 
of  resources  expended  during  both  the  development  operation 
of  the  system.  For  system  development,  typical  ratios 
would  involve  resource  requirements  for  manpower  and  cap¬ 
ital.  For  manpower  requirements  it  is  useful  to  consider 
resources  at  various  skill  levels  such  as  clerical,  pro¬ 
grammer,  and  systems  analyst  for  various  specific  tasks. 

The  ratios  of  actual  usage  to  planned  usage  of  these  per¬ 
sonnel  resources  will  help  to  facilitate  planning  of 
personnel  requirements  for  future  systems  work  if  the 
tasks  axe  sufficiently  described  and  standardized. 

Resource  requirement  ratios  for  the  operation  of  a 
system  should  not  be  limited  to  those  applied  to  run  the 
system,  such  as1 manpower  and  supplies.  Resources  internal 
to  the  system  should  also  be  considered  relative  to  the 
hardware  and  software  capability  absorbed  by  each  system. 
Examples  are  capacity  utilization  of  mass  storage,  CPU 
time  and  peripheral  or  I/O  devices. 

To  measure  the  current  utilization  of  computer  re¬ 
sources,  a  variety  of  hardware  and  software  monitoring 
tools  ape  available.  These  tools  arc  especially  useful 
for  measurement  of  time  ot  capacity.  The  system  itself 
will  often  provide  outputs  which  indicate  the  current 
level  of  utilization  of  such  resources.  . 

Information  regarding  actual  utilization  of  personnel, 
consumables  and  capital  can  be  found  in  project  management, 
operational  and  accounting  records.  Where  these  are  not 
available,  it  may  be  cost  justified  to  observe  system  per¬ 
formance  to  measure  critical  resource  utilization. 

Comparison  is  impossible  unless  target  or  budgeted 
requirements  are  documented  by  management  and  changes  in 
needs  are  carefully  included.  These  records  must  be  spe¬ 
cific  about  well-defined  tasks  and  activity  to  be  meaning¬ 
ful. 
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•  Result:  This  phase  of  the  procedure  produces  a 
documented  comparison  of  planned  resource  requirements 
with  actual  resource  utilization.  These  comparisons  are 
normally  expressed  in  terms  of  ratios  which  indicate  the 
relative  level  of  performance.  These  statistics  should 
be  gathered  for  different  time  periods  to  assess  trends. 
Care  should  be  taken  in  interpreting  needs  during  transi¬ 
tional  periods, such  as  installation. 

For  ratios  of  significant  resources  which ‘ indicate 
unusual  variance  from  norms,  the  team  should  report  steps 
taken  to  find  reasons  for  the  discrepancy.  Frequently, 
changes  in  original  objectives  or  specifications  have  not 
been  factored  into  the  expected  resource  requirements. 

d.  Sensitivity  of  System  Output 

•  Purpose:  To  determine  the  effect  of  variations  in 
data  input  or  assumed  environment  on  the  success  of  the 
system.  The  sensitivity  analysis  determines  the  likely 
magnitude  and  character  of  changes  in  system  output  and 
their  subsequent  impact  on  user  behavior.  These  changes 
may  result  from  normal  fluctuation  of  input  values,  from 
input  error,  from  suppression  or  filtering  of  input,  from 
changes  in  source,  from  time  period  inconsistencies,  from 
system  alteration,  and  from  changes  in  the  functional  or 
system  environment. 

Depending  on  the  expectation  for  variation  in  these 
areas,  the  ability  of  the  system  to  adapt  to  change  may  be 
a  highly  desirable  or  even  necessary  asset.  On  the  other 
hand,  wide  variations  of  input  may  have  no  significant 
effect  on  user  performance  or  selection  of  alternatives. 
The  latter  suggests  that  the  system  may  have  little  value 
for  anything  other  than  execution  and  some  operational 
activities. 
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Selection 


Characteristics 


Planning 

Management 

Communication 

Data  Base 

Reference 

Monitor 

Scientific 


Relevant  and  advisable 
Relevant  and  necessary 
Rarely  relevant 
Relevant  and  advisable 
Relevant  and  advisable 
Relevant  and  usually  necessary 
Relevant  and  necessary 


Activity  Levels  —  Strategic,  tactical  and  often 
opera tio, ^1 


Methods  —  Naming;  simulation;  graphical  analysis; 
work  sampling 

•  Conduct:  Problems  associated  with  this  phase  of 
the  analysis  stem  from  the  fact  that  measurement  is  not 
always  possible  if  the  system  is  currently  constrained  by 
active  use.  Thus,  in  addition  to  usual  measurement  prob¬ 
lems,  the  evaluation  team  also  faces  the  problem  of  hy¬ 
pothesizing  and  imposing  likely  changes  to  the  environment 
or  the  input  stream.  Earlier  phases  of  the  analysis  (es¬ 
pecially  A3)  should  provide  a  good  background  in  the  general 
types  of  changes  to  anticipate,  but  interpretation  of  these 
in  terms  of  the  system  may  bt  difficult.  Gaming  techniques 
may  be  useful  in  determining  the  user’s  modified  system 
needs  when  changes  are  hypothesized. 

In  some  cases,  it  will  be  possible  to  use  the  system 
under  changed  conditions.  Work  sampling  cm  then  be  used  to 
study  the  resulting  effects  on  system  output1-  and/or  user 
behavior.  Systems  based  on  certain  specific  mathematical 
models  provide  the  potential  to  gather  sensitivity  informa¬ 
tion  as  a  direct  consequence  of  applying  the  algorithm. 
Relationships  among  inputs,  outputs  and  probable  alterna¬ 
tives  of  action  can  also  be  depicted  by  graphical  metko ’s 
to  show  the  possible  changes  in  user  behavior. 
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Probably  the  most  widely  useful  technique  for  sen¬ 
sitivity  analyses  is  simulation.  Actual  experimentation 
with  the  system  may  be  infeasible  because  of  ongoing  opera¬ 
tions  or  because  of  economic  considerations.  Simulation 
facilitates  sensitivity  testing  under  a  wide  variety  of 
conditions  and  input  factors.  The  main  problem  with  sirr”  - 
lation  is  ensuring  that  all  critical  factors  of  the  real 
system  are  included  in  a  valid  simulation  model.  This 
approach  allows  controlled  experimentation  to  assess  the 
statistical  significance  of  changes  in  the  parameters  and 
factors  comprising  the  simulation  model. 

•  Result:  Conclusions  of  the  sensitivity  analysis 
should  be  included  in  a  formal  report.  This  report  should 
describe  the  types  of  changes  which  the  system  can  accom¬ 
modate  and  the  boundaries  within  which  th^se  changes  cause 
no  major  alteration  of  output  significance.  Also,  expected 
changes  which  will  alter  substantially  or  deteriorate  either 
system  performance  or  user  function  should  be  indicated. 

If  little  change  is  possible  or  system  output  is  impervious 
to  change,  the  value  and  cost  of  the  system  should  be  made 
commensurate  with  the  highly  restricted  potential  benefits. 

e.  Impact  on  Other  Systems/Functions 

•  Purpose:  To  measure  the  effect  that  this  system 
exerts  on  related  systems  and  functions.  The  net  worth  of 
the  system  may  be  impossible  to  measure  if  considered  as 
an  isolated  entity.  In  such  cases,  this  phase  of  the 
analysis  assumes  a  great  deal  of  importance. 

•  Selection: 


Characteristics 


Planning 
Management 
Communication 
Dat.i  Base 


Desirable;  usually  relevant 
Relevant  and  necessary 
Relevant  and  necessary 
Relevant  and  necessary 
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Reference  Relevant  arid  necessary 

Monitor  Rarely  relevant 

Scientific  Rarely  relevant 

Activity  Levels  —  Tactical  and  operational; 
strategic  in  its  relation  as  a  by-product  of  other 
systems;  execution  in  its  relation  as  a  source  of  data 
to  others. 

Methods  —  Accounting  records;  consensus;  simulation 

•  Conduct:  This  phase  is  particularly  important  in 
situations  where  several  systems  serve  the  same  functional 
activity  or  where  new  systems  are  introduced.  Assuming 
that  the  effect  on  key  performance  indicators  can  be  mea¬ 
sured,  it  will  be  difficult  to  determine  the  exact  cause 
of  any  change.  It  may  be  possible  to  ascertain  the  credit 
due  to  a  system  by  accounting  methods  or  p re -determined 
criteria.  In  any  event,  it  is  normally  necessary  to  so¬ 
licit  subjective  opinions  about  the  relative  contribution 
of  system  or  environmental  changes  to  differences  in  func¬ 
tional  performance.  The  consensus  may  be  used  as  the  basis 
of  an  accounting  allocation  or  of  a  qualitative  judgment 
of  system  worth.  If  sophisticated  models  of  the  organiza¬ 
tion  exist,  it  may  be  possible  to  assess  the  cause  and  ef¬ 
fect  relationships  by  a  series  of  experiments.  The  latter 
approach  is  normally  too  costly  for  such  an  imprecise 
estimate. 

-  Result:  The  benefits  and  costs  assigned  to  a  given 
system  must  bo  modified  in  light  of  its  impact  on  other 
proximate  systems  or  its  contribution  to  functions  relative 
to  other  systems.  This  phase  reports  the  proportion  of 
direct  benefits  and  the  absolute  value  of  any  indirect  ben¬ 
efits  which  can  be  legitimately  (with  management  concur¬ 
rence)  claimed  by  the  system.  In  addition,  any  costs 
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incurred  by  other  systems  or  functional  overhead  which 
can  be  attributed  to  the  system  under  study  are  identified 
as  supplements  to  costs  considered  earlier  in  the  evalua¬ 
tion  study. 
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IV.  Summary 


This  manual  is  a  working  document  to  assist  in  evaluating 
a  variety  of  information  processing  activities.  A  generalized 
procedure  is  outlined  to  assess  the  capability  of  an  information 
system  to  meet  functional,  user-oriented  requirements.  The 
authors  propose  a  composite  overview  which  will  be  the  basis  of 
a  systematic  approach  to  information  system  evaluation.  Stress 
is  placed  upon  identification  of  tangible  benefits  and  measures 
of  effectiveness.  Refinement  aM  standardization  of  the  approach 
are  anticipated  on  the  basis  of  experience  in  applying  its  phases 
to  systems  operating  under  differing  circumstances. 
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APPENDIX  A 
Glossary 


Activity  levels  —  Categories  of  functional  activity  and  tasks 
which  involve  varied  degrees  of  decision-making  and  opera¬ 
tional  involvement.  See  Table  1-1. 

Characteristics,  Processing  —  Categories  of  data  processing  ac¬ 
tivity  designated  by  fundamental  nature  of  handling,  storage, 
operation,  and  dissemination  within  a  system.  See  Table  1-2. 

Decision  System  —  The  set  of  decision  rules  which  are  integral 
to  a  functional  activity  in  an  organization.  Information  must 
be  provided  to  employ  the  rules  and  the  decision  system  spec¬ 
ifies  the  supporting  information  system(s). 

Delphi  Technique  —  A  systematic,  iterative  means  of  gathering  and 
aggregating  subjective  opinions  about  uncertain  or  speculative 
situations  from  a  group  of  knowledgeable  persons.  Avoids  bias 
normally  associated  with  face  to  face  encounters  among  partic¬ 
ipants  . 

Functional  Activity  —  Activity  directly  related  to  the  functions 
which  must  be  performed  in  order  to  achieve  organizational  ob¬ 
jectives  or  mission. 

Gaming  —  An  exercise  in  which  humans  play  roles  in  a  contrived 
typical  or  actual  environment  under  controlled  conditions. 

The  players  are  confronted  with  competitive  or  conflict  situa¬ 
tions  which  include  chance  results  from  performed  tasks.  The 
°nvironment  and  situations  are  described  in  scenarios.  Pur¬ 
poses  of  evaluation  emphasize  operational  gaming  rather  than 
training,  diagnosis,  experimentation,  or  entertainment. 

In-basket  Analysis  —  A  gaming  approach  which  involves  reaction  of 
an  individual  to  a  series  of  planned  conflict  situations  rep¬ 
resented  by  documents  delivered  to  him.  A  scenario  establishes 
the  role  of  the  individual,  background  facts,  and  the  opera¬ 
tional  circumstances  for  which  the  documents  are  typical. 
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Alternatively,  the  analysis  may  refer  to  sampling  of  the 
documents,  and  their  sources  and  destinations,  which  are  en¬ 
countered  by  an  individual.  This  provides  a  description  of 
activity  and  information  flows  of  the  individual.  It  may  be 
used  as  representative  gaming  material  for  his  circumstance 
and  position. 

Information  service  or  support  —  Assistance  required  or  provided 
by  a  group  which  involves  transfer  of  information.  This  may 
utilize  a  number  of  sources  or  systems  to  provide  the  service 
or  support. 

Information  system  —  A  set  of  interrelated  rules  and  procedures 
for  processing  data  into  information  in  order  to  get  or  con¬ 
trol  action. 

Measure  of  effectiveness  (MOE)  —  Parameter  or  variable  which 
indicates  the  level  of  performance  in  an  essential  aspect  of 
a  system.  Usually  expressed  in  tangible  terms  related  directly 
to  functional  objectives  and  effects. 

Methods  of  analysis  —  Variety  of  techniques  which  can  be  utilized 
to  study  characteristics  and  activity  of  a  system  in  either 
quantitative  or  qualitative  terms.  See  Table  II -1. 

Performance  measurement  --  Analysis  of  computer  system  performance; 
normally  in  terms  of  speed  of  operations,  computer  throughput, 
capacity  utilization,  response  time  and  other  factors  related 
to  the  computer  itself. 

Requirements  —  Functional  description  of  a  system(s)  expressed  in 
terms  of  the  needs  of  the  user  or  customer  of  the  system  and 
its  output.  (See  Appendix  B.) 

Scenario  —  Description  of  the  environment,  tasks,  and  roles  asso¬ 
ciated  with  an  operational  situation.  Normally  shows  the  setting 
and  scripts  of  behavior  related  to  actual  use  or  operation  of 
functional  activities  or  information  systems.  Since  it  is  based 
upon  current  experience,  it  can  be  contrasted  with  intended 
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procedures  or  reactions.  Often  includes  instructions  for 
gaining  participants  under  varied  conditions  and  for  those  who 
conduct  or  assess  the  performances. 

Simulation  —  Process  of  generating  representative  numerical  re¬ 
sults  by  using  a  model  of  the  actual  process  or  activity. 
Normally  used  when  uncertainty  or  comple/.  interrelationships 
make  analytical  solutions  infeasible.  Distinguished  from 
gaming  which  uses  a  simulated  (modelled)  environment  to  de¬ 
velop  human  behavioral  reactions  and  output. 
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APPENDIX  B 


System  Conceptualization  and  Definition 
1.  Preliminary  Phases 

a.  Problem  Investigation.  Customer  exploration  of  ideas 
about  new  or  revised  applications  or  systems.  This  includes  ini¬ 
tial  formulation  of  project  requests,  evaluation  of  available  re¬ 
sources  to  solve  a  customer  problem,  and  development  of  proposed 
solutions  to  customer  problems.  This  may  replace  or  be  included 

in  feasibility  studies  for  small  or  well-defined  problems.  Results 
in  a  formal  Problem  Statement  and  leads  toward  economic  analysis 
and  Justification  of  a  proposed  approach  to  problem  solution. 

b.  Feasibility  Study .  A  study  for  management  analysis,  pre¬ 
pared  primarily  by  systems  analysts,  to  assess  the  technical, 
operational  and  economic  issues  related  to  achieving  a  stated  cus¬ 
tomer  objective.  For  simpler  applications,  economic  analysis  for 
Project  Justification  in  the  Problem  Investigation  phase  may  be 
adequate.  Most  systems  will  require  a  formal  study  and  report, 
consisting  of  the  following  general  content: 

•  Objective. 

•  Preliminary  statement  of  requirements  and  design 

features . 

•  Alternative  approaches. 

•  Recommended  solution. 

•  Justification,  priority  and  benefit  analysis. 

•  Scheme  for  implementation. 

•  Estimates  of  resources. 

-  Personnel 

-  Equipment 

-  Financial 

•  Organizational  effects  and  support. 

•  Task  force  or  project  team  requirements  and  possible 

composition. 

•  Preliminary  project  plan,  including  overall  networks 

and  major  task  definition. 
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2 .  User  Requirements .  A  functional  description  of  the  system 


prepared  by  customer  representatives  and  system  analysts  subse¬ 
quent  to  feasibility  study  and  development  of  priority  assignment. 

The  User  Requirements  should  emphasize  a  new  system  and  its  char¬ 
acteristics  rather  than  describe  an  existing  one.  If  substantial 
differences  with  the  Feasibility  Study  appear,  it  may  be  necessary 
to  reevaluate  the  economic,  operational  and  technical  justification. 

This  phase  is  dedicated  to  system  analysis  and  to  user  orientation 
and  understanding.  Facts  must  be  collected  about  the  current  and 
proposed  system  through: 

a.  User  interview. 

b.  Operations  observation  and  activity  analysis. 

c.  Source  data  identification. 

d.  Organization  review  and  analysis. 

Correlations  with  d-ta  in  other  systems  and  identification  of  crit¬ 
ical  outputs  must  be  accomplished.  The  processing  relationships 
must  be  established  and  the  possible  impact  of  the  new  system  on 
affected  organizational  units  must  be  indicated.  The  Requirements 
should  be  documented  into  a  formal  report,  subject  to  customer  ap¬ 
proval,  with  the  following  content: 

•  Mission  and/or  problem  statement. 

•  System  Objectives. 

-  Management  agreement  with  scope  of  system 

-  Establish  criteria  for  success 

-  Determine: 

a)  primary  functional  and/or  organizational 
units  served  by  the  system 

b)  secondary  functional  and/or  organizational 
units  served  by  the  system 

c)  inter-intra  organizational  operation  rela¬ 
tionships 

-  Establish  a  glossary  of  terms  pertinent  to  the 

system. 

•  Output  and  Reporting  Definition. 
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-  Determine  management  reports  required  by  the 

business,  independent  of  cost  of  supplying  such 
reports. 

-  Establish  a  rough  cost/value  relationship  of  each 

management  report. 

-  Define  terms  and  fields  for  each  report. 

-  Identify  report  quantity,  distribution,  and 

schedule  constraints. 

-  Ascertain  security  and/or  proprietary  aspects  of 

each  report. 

-  Identify  influences  on  report  requirements: 

a)  Government  regulations 

b)  Company  auditors 

c)  Public  accountants  and  auditors 

d)  Customer  requirements 

-  Evaluate  possibilities  and  value  of  improved 

customer  service. 

•  Input  Definition. 

-  Determine  input  data  requirements  based  on  anal¬ 

ysis  of  output. 

-  Determine  files  in  which  such  data  already  exist 

(these  files  may  be  "common"  to  more  than  one 
system) . 

-  Establish  sources  of  data  that  may  have  to  be 

gathered  from  new  fox'ms  or  procedures. 

-  Outline  data  preparation  instructions. 

-  Study  volumes : 

a)  Normal 

b)  Peak 

c)  Frequency  of  peaks 

d)  Expansion  allowance 

•  File  Definition 

-  Define  content  and  purpose. 

-  Estimate  size,  form  and  rate  of  expansion. 

-  Describe  record  types  and  storage  media. 

-  Define  maintenance,  updating,  access  activity. 

-  Outline  procedures  to  afford  and  validate  access. 

-  Define  file  retention  constraints. 
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a)  Legal  restrictions 

b)  Functional  requirements,  e.g.,  closed 
charge  numbers,  inactive  name  and  address 
files,  etc. 

Controls  and  Tests. 

-  Indicate  edits  for  data  validity  and  reasonable¬ 

ness. 

-  Specify  disposition  of  data  rejected  by  edits. 

-  Establish  requirements  for  control  totals,  hash 

totals,  record  counts,  programmer  or  operator 
access  restrictions,  checkpoints  or  restarts, 
etc. 

-  Determine  backup  and  recovery  requirements  in 

case  of  file  loss  or  damage,  or  catastrophic 
error. 

-  Indicate  test  requirements,  quality  assurance 

needs  and  sensitivity  to  system  performance. 

Schedule  constraints 

-  Describe  operating  cycle  and  turnaround  character¬ 

istics. 

-  Prescribe  capability  and  priority  commitments. 

-  Analyze  fluctuations  and  cycles  of  activity. 

-  Effect  of  schedule  delays  on  system  and  related 

systems. 

-  Define  requirements  for  the  system  to  "carry-on" 

on  a  manual  basis  if  computer  is  down  for  a 
prolonged  period. 

Cost  Analysis  and  Resource  Constraints 
Proposed  Frojeet  Plan 
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APPENDIX  C 

Computer  System  Factors 


A.  Organisational  Environment 

1.  Fiscal  operations  procedures  and  control  of  the  computing 
service  department. 

2.  Personnel  qualifications  and  organizational  sti'ucture 

3.,  Processing  standards,  controls  and  security  of  program 
and  data  files 


B.  •  Application  Description 


1.  Dominant  Operations 

Identify  the  nature  and  extent  of  each: 

a.  I/O,  user  and  system  communication 

b.  Editing,  reformating,  error  checking 

c.  Computation 

d.  File  create,  update,  restructure 

I 

2.  Storage 

a.  File  name  or  category 

b.  Contents  --programs,  data,  queries,  purpose 

c .  Medium 

d.  File  organization  —  access  requirements,  key  fields 

e.  File  size  —  current  or  expected,  growth  allowance 

f.  Record  structure  --  fixed,  variable  length,  codification 


3.  Input 


a.  Input  message  name  or  category 

b.  Data  typo  —  Dijital,  analog,  alphanumeric,  text 

c.  Content,  purpose,  descriptive 

d.  Format  —  rigid,  free,  special  requirements,  restrictions 

e.  Entry  mode 

(1)  Transcribed  centrally 
--keyboard 

— optical  scan 
— me  lia  conversion 

(2)  Transcribed  at  work  site 
— portable  keyboard 

--  paper  tape  punch 
--magnetic  tape  recorders 

(3)  Direct  entn  from  remote  work  site 
— analog  -  Digital 
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— graphical  display  with  light  pen  (etc.) 
—OCR 

— interactive  keyboard 
f.  Transaction  frequency  and  volume 

(1)  Intermittent  Single,  intermittent  Bursts, 
Continuous 

(2)  Normal  entry  rate  current 
(2)  Peaks  entry  rate  current 

(4)  Peaks  entry  rate  maximum  growth 

For  (2),  (3),  and  (4),  estimate  mean,  range 
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Output 


a.  Output  message  name  or  category 

b.  Data  media  —  hard  copy,  graphical,  voice,  machine 

c.  Contents,  purpose,  description 

d .  Fo  rma  t  re  qui rement  s 

e.  Reporting  mode 


Scheduled 
By  exception 

Background  (as  time  available) 
Articled  inquiries  ^  ±  , 

Spontaneous  inquiries  v 

Browsing  or  tutorial 


f.  Output  frequency  and  volume 

(1)  Normal  rate  current 

(2)  Peak  rate,  current 

(3)  Peak  rate,  maximum  growth 
Estimate  mean,  range 


C.  Computing  System  Attributes 

1.  Which  of  the  following  system  attributes  are  existing 
problems  for  your  application? 

2.  Which  are  apt  to  become  as  problems  with  anticipated  growth? 

3.  What  additions  or  changes  to  the  system  would  help  alleviate 
the  problem? 


Program  Development  Cycle 

Hardware  independence  -  ability  to  develop  and  execute  programs  with 
varying  hard-ware  configurations 

Development  of  new  programs  (adequacy  of  programming  languages  and 
debug  tools) 

Standard  Library  routines 
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Modularity  o f  programs 

Hodif ication  and  maintenance  of  programs 

Adaptation  and  parameterization  of  programs 

Preparation  and  generation  of  test  eases  and  materials 

Verif ication  of  test  results 

Program  Operations  Cycle 

Impact  on  existing  applications 

Turnaround  time 

Accuracy  an!  verification 

Frequency  of  use 

Schedule  of  execution 

Transfer  of  data  among  systems 

Sensitivity  to  job  mix 

Common  processing  standards 

Compatibility  of  programs,  data,  vocabulary 

Reliability  and  backup  of  equipment 

Accommodates  sources 

Data  Ease  Upkeep  Cycle 

Data  Preparation  Activity 
Modularity  ol  files 
Ease  of  updating  files 
Common  recording  standards 

D.  Computing  Environment 

1.  Hardware  Description 

a.  Processing  units  (CPU) 

(1)  How  many  processing  units?  Do  they  perform 
specialized  functions?  Identify  the  nature  ol  each. 

(?)  Support  lor  fixed  point  arithmetic  -  word  sizc(s) 

floating  point  arithmetic  -  word  size(s) 
Decimal  arithmetic  -  word  size(s) 

(1)  Describe  special  hardware  provisions  for 
--interrupt  haniling 
--projram  relocation  and  rddressing 
— detecting  equipment  rralf  ructions 
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— modularity  and  configuration  of  hardware 
components 

— protection  of  data  integrity 
— storage  access 
— emulation,  microprogramming 
— multiprocessing 

b.  Storage  -  For  each  type,  identify  manufacturer,  name 
of  unit,  how  many  you  have,  how  many  can  be  supported, 
and  storage  capacity  per  unit  (in  bits).  Describe  any 
special  features. 

(1)  Core  storage 

High  speed  main  storage 
Slower  speed  mass  storage 

(2)  Sequential  access 
Magnetic  tape 

(3)  Direct  access 

Disk 

Drum 

Magnetic  cards 

c.  I/O  Communication 

For  each  type,  identify  manufacturer,  name  of  unit, 
how  many  can  be  supported.  Describe  any  special 
features . 

(1)  Unit  record  equipment 

Card  Reader'/Punch 
Paper  Tape  Reader/ Punch 
Magnetic  character  reader 
Line  pi  inter 

(2)  Interactive  Communication 

Printer/Keyboard  -  TTY  or  I/O  Typewriter 
Graphics  -  keyboard  and  CRT 
Voice  input  and/or1  response 
Special  purpose  console 

(3)  Intersystem  Communication 

Phono  lines 
TTY 

Special  purpose 

(4)  Others 
Plotters 

Optical  Scanners 
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d.  Configuration  of  a,  b,  and  c 


2.  Support  Software  Description 

For  each  category  of  system  support,  .list  which  services 
are  available  under  the  system,  describe  the  service  briefly,  and 
indicate  whether  you  have  it.  Following  each  category  below  are 
some  key  features  to  consider  in  the  description. 

a .  Task  Execution  Support  Services 

Multiprogrommed?  Fixed  or  variable  number  of  tasks? 
Maximum  number  of  tasks?  Multiprocessing? 

Local  batch,  remote  Latch,  interactive  modes. 

b.  Programming  Languages 
What  languages? 

Available  in  what  modes;  local  or  remote  batch, 
interactive? 

Compile,  Assemble,  Interpret,  Generate 

c .  Data  Management  Methods 

Describe  in  terms  of  file  organization  requirements. 

d .  Debug  and  Test  Facilities 

Program  tracing,  dumps 
Test  da  ta  p r ( pa ra t ion 

e.  Progr-m  Library  Facilities 

Creation  and  up'.'te  capabilities 
Nature  rni  SUe  of  Library 

f .  Support  Projrams 

Utility,  data  conversion,  test  generators,  debugging 
aids,  edit  features 
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